Surgical visualization systems and displays

ABSTRACT

A medical apparatus is described for providing visualization of a surgical site. The medical apparatus includes an electronic display disposed within a display housing. The medical apparatus includes a display optical system disposed within the display housing, the display optical system comprising a plurality of lens elements disposed along an optical path. The display optical system is configured to receive images from the electronic display. The medical apparatus can include.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority under 35 U.S.C. 119(e)to U.S. Provisional Application No. 62/138,351, entitled “SURGICALVISUALIZATION SYSTEMS AND DISPLAYS,” filed on Mar. 25, 2015 U.S.Provisional Application No. 62/138,919, entitled “SURGICAL VISUALIZATIONSYSTEMS AND DISPLAYS,” filed on Mar. 26, 2015; U.S. ProvisionalApplication No. 62/183,148, entitled “SURGICAL VISUALIZATION SYSTEMS ANDDISPLAYS,” filed on Jun. 22, 2015; U.S. Provisional Application No.62/184,222, entitled “SURGICAL VISUALIZATION SYSTEMS AND DISPLAYS,”filed on Jun. 24, 2015; U.S. Provisional Application No. 62/184,838,entitled “SURGICAL VISUALIZATION SYSTEMS AND DISPLAYS,” filed on Jun.25, 2015; U.S. Provisional Application No. 62/187,796, entitled“SURGICAL VISUALIZATION SYSTEMS AND DISPLAYS,” filed on Jul. 1, 2015;and U.S. Provisional Application No. 62/260,221, entitled “SURGICALVISUALIZATION SYSTEMS AND METHODS WITH EXOSCOPES,” filed on Nov. 25,2015. The entirety of each application referenced in this paragraph isincorporated herein by reference.

BACKGROUND Field

Embodiments of the present disclosure relate to visualization systemsand displays for use during surgery.

Description of Related Art

Some surgical operations involve the use of large incisions. These opensurgical procedures provide ready access for surgical instruments andthe hand or hands of the surgeon, allowing the user to visually observeand work in the surgical site, either directly or through an operatingmicroscope or with the aid of loupes. Open surgery is associated withsignificant drawbacks, however, as the relatively large incisions resultin pain, scarring, and the risk of infection as well as extendedrecovery time. To reduce these deleterious effects, techniques have beendeveloped to provide for minimally invasive surgery. Minimally invasivesurgical techniques, such as endoscopy, laparoscopy, arthroscopy,pharyngo-laryngoscopy, as well as small incision procedures utilizing anoperating microscope for visualization, utilize a significantly smallerincision than typical open surgical procedures. Specialized tools maythen be used to access the surgical site through the small incision.However, because of the small access opening, the surgeon's view andworkspace of the surgical site is limited. In some cases, visualizationdevices such as endoscopes, laparoscopes, and the like can be insertedpercutaneously through the incision to allow the user to view thesurgical site.

The visual information available to a user without the aid ofvisualization systems and/or through laparoscopic or endoscopic systemscontains trade-offs in approach. Accordingly, there is a need forimproved visualization systems, for use in open and/or minimallyinvasive surgery.

SUMMARY

Disclosed herein are systems, devices, and methods for surgery andsurgical visualization and display. Image acquisition and image display,for example, are described. Such image acquisition may be performed by,such as for example but not limited to, one or more cameras on asurgical tool, frame or support just a few centimeters above thepatient's body and/or surgical site, as well as camera systems fartherfrom the patient including camera systems from about 15 cm to about 45cm from the patient's body and/or the surgical site. In variousembodiments, these cameras may be stereo or mono cameras. A variety ofcamera designs may be employed. Different types of displays and displaydesigns including binocular displays may also be used. Variouscombinations of components and features are possible. For example, oneor more embodiment or feature described or referenced in any one or moreof the different sections of the present disclosure may be used with,combined with, incorporated into, and/or are otherwise compatible withone or more embodiments and features described in any one or more otherof the sections of the present disclosure. Similarly, embodiments orfeatures described or referenced in any section of the presentdisclosure may be used with, combined with, incorporated into, and/orare otherwise compatible with any other embodiment or feature alsodescribed or referenced in that section. Additionally any one or moreembodiments or features described or referenced in any section may beused with, combined with, incorporated into, be applicable to, and/orare otherwise compatible with a wide range of medical or surgicaldevices which may or may not be introduced into the body including butnot limited to endoscopes, laparoscopes, and arthroscopes. Use of thevarious features and embodiments and combination thereof with othermedical devices is also possible.

In certain embodiments, a visualization system is provided. Thevisualization system comprises a plurality of communication portsconfigured to be operatively coupled to a plurality of image acquisitionsubsystems and a lighting system comprising one or more light sources;at least one image output port configured to be operatively coupled toat least one image display subsystem; at least one user input portconfigured to be operatively coupled to at least one user input device;and at least one circuit operatively coupled to the plurality ofcommunication ports, the at least one image output port, and the atleast one user input port. The at least one circuit is configured toreceive data signals from the plurality of image acquisition subsystems,to transmit control signals to the lighting system, and to transmitoutput image signals to the at least one image display subsystem. The atleast one circuit is further configured to receive at least one firstuser input signal from the at least one user input device, the at leastone circuit responsive at least in part to the received at least onefirst user input signal by: selecting an image acquisition subsystemfrom the plurality of image acquisition subsystems, transmitting theoutput image signals to the at least one image display subsystem inresponse to the data signals received from the selected imageacquisition subsystem, and generating and transmitting the controlsignals to the lighting system. The lighting system is configured torespond to the control signals by switching light intensity settings ofthe lighting system for the selected image acquisition subsystem.

The systems, methods and devices described herein each have innovativeaspects, no single one of which is solely responsible for the desirableattributes disclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

Throughout the drawings, reference numbers can be reused to indicategeneral correspondence between reference elements. The drawings areprovided to illustrate example embodiments described herein and are notintended to limit the scope of the disclosure.

FIG. 1 illustrates an embodiment of the surgical visualization systemhaving an imaging system that can be configured to provide imagerysimilar to a direct-view surgery microscope.

FIG. 2 illustrates an example surgical viewing system attached to anarticulating arm, the system including one or more cameras mounted on abinocular viewing platform.

FIGS. 3A and 3B illustrate an example surgical viewing system thatincludes an isocenter positioning system attached to the binocularviewing platform.

FIGS. 4A and 4B illustrate an embodiment of a surgical visualizationsystem having an optical imaging system mounted under the binocularviewing platform.

FIG. 5A illustrates an example embodiment of an optical imaging systemfor use in a stereoscopic surgical viewing system, such as thoseillustrated in FIGS. 4A and 4B.

FIG. 6A is a front view of an embodiment of a surgical visualizationsystem, a movement control system, and an imager.

FIG. 6B is a front view of the embodiment of FIG. 6A with the movementcontrol system and imager shifted.

FIG. 6C is a partial section view of the embodiment of a movementcontrol system of FIG. 6A.

FIG. 7 is a side view of an embodiment of a surgical visualizationsystem, a movement control system, and an imager.

FIG. 8 is a rear view of an embodiment of an embodiment of a movementcontrol system.

FIG. 8A illustrates a perspective view of an example gimbal system foran imager, the gimbal system coupled to a viewing assembly comprisingtwo pairs of oculars.

FIG. 8B illustrates a perspective view of a second example gimbal systemfor an imager, the gimbal system coupled to a viewing assembly.

FIGS. 9A-9B illustrate example display optical systems configured toprovide a view of a display or a pair of displays through oculars.

FIG. 10 is a schematic illustration of a surgical visualization systemincluding an assistant display.

FIG. 11 schematically illustrates an example medical apparatus inaccordance with certain embodiments described herein.

FIGS. 12A-12C schematically illustrate another example medical apparatusin accordance with certain embodiments described herein.

FIG. 13A illustrates a schematic of an example of a composite image witha picture-in-picture (PIP) view of a surgical field.

FIG. 13B schematically illustrates a front view of an embodiment of amedical apparatus incorporating left and right assemblies to produce acomposite image of two or more images for both left and right eyes.

FIG. 13B1 shows an illustration of an example medical apparatusaccording to certain embodiments described herein.

FIG. 13B1-a shows a larger view of the side view of FIG. 13B1.

FIG. 13B1-b shows a larger view of the front view of FIG. 13B1.

FIG. 13B1-c shows the example medical apparatus of FIG. 13B1.

FIG. 13B1-d shows an illustration of a beam combiner, a first camera,and a second camera.

FIGS. 13B1-e and 13B1-f illustrate example display units and opticalpaths of a medical apparatus.

FIG. 13C illustrates a schematic of an example view of multiple imagesof a surgical field combined adjacent to one another.

FIG. 14A shows a schematic of an example medical apparatus comprising aframe 510 disposed above a surgical site of a mock patient.

FIG. 14A1-a, FIG. 14A1-b, and FIG. 14A-c schematically illustratescameras mounted to a circular frame, a square frame, or a L-shaped framerespectively.

FIG. 14B1-a shows an illustration of an imaging system comprising acamera, fiber optics, and a laparoscope.

FIG. 14B1-b shows an illustration of certain embodiments of a medicalapparatus having one or more proximal camera D on a frame.

FIG. 14B2-a schematically illustrates imaging optics of an exampleimaging system compatible with certain embodiments of cameras asdescribed herein.

FIG. 14B2-b shows an illustration of an example top-down view of certainembodiments disclosed herein.

FIG. 14B2-c shows an illustration of an example side-view of one opticalchannel of the apparatus shown in FIG. 14B2-b.

FIG. 14B2-d shows an illustration of an example proximal cameraarrangement in accordance with certain embodiments described herein.

FIG. 14B2-e schematically illustrates a display viewable throughportals.

FIG. 14B2-f shows an illustration of an example planar four-barmechanism.

FIG. 14B2-g shows an illustration of the side-view of FIG. 14B2-f.

FIG. 14B3 shows an illustration of an oblique camera orientation.

FIGS. 15-21 schematically illustrate examples of a medical apparatusthat utilize a mobile display device in accordance with certainembodiments described herein.

FIG. 22 shows an example medical apparatus having multiple displayswithin the field of view of an ocular.

FIG. 23 illustrates an example embodiment of a mobile display device incommunication with a docking port disposed on a binocular display unit.

FIGS. 24-25 illustrate example headsets or head mounted displays used toview images from a mobile display device.

FIGS. 26-30 illustrate various example display optics designs inaccordance with certain embodiments described herein

FIG. 31 illustrates a laser distance positioning guide in accordancewith certain embodiments described herein.

FIGS. 32-33 illustrate example embodiments on producing multiple imagesin different wavelength bands.

FIG. 34 illustrates an example optical imaging system or camera inaccordance with certain embodiments described herein

FIG. 35 illustrates various embodiments for fluorescence imaging.

FIG. 36 illustrates various embodiments for narrow band imaging.

FIGS. 37A-B illustrates an example ergonomically beneficial binocularviewing assembly for providing a surgeon with video of a surgical site.

FIGS. 38-39 illustrate an example ergonomically beneficial binocularviewing display having oculars beneath corresponding displays.

FIG. 40 illustrates an example surgical visualization system thatincludes a primary surgeon camera and an assistant camera that isrotatable around a central aperture.

FIG. 41 illustrates an example binocular viewing assembly that includessurgeon oculars and assistant oculars.

FIG. 42 illustrates an example optical imaging system for providing asurgical microscope view of a surgical site.

FIG. 43 illustrates an example optical imaging system for providing asurgical microscope view of a surgical site using a common objective forleft and right optical paths, wherein each optical path is split to atleast two image sensors.

FIG. 44 illustrates an example of a low profile surgical microscopecamera assembly including a housing.

FIG. 45 illustrates an example visualization system with two binoculardisplay units configured for multi-view switching.

FIG. 46 illustrates an example binocular viewing display unit withcamera controls integrated with a handle.

FIG. 47 illustrates an example embodiment of an electro-mechanicalcircuit diagram that has multiple switches which control one function.

FIG. 48 illustrates an example algorithm that demonstrates how a smallset of buttons can be configured to operate a larger number offunctions.

FIG. 49 schematically illustrates an example of an imaging system in asimplified operating room configuration.

FIG. 50A schematically illustrates an example visualization systemcontroller in accordance with certain embodiments described herein.

FIG. 50B schematically illustrates a partial view of an examplevisualization system controller operatively coupled to a first imageacquisition subsystem comprising a first camera and to a second imageacquisition subsystem comprising a second camera in accordance withcertain embodiments described herein.

FIG. 50C schematically illustrates a partial view of an examplevisualization system controller operatively coupled to first and secondimage acquisition subsystems each comprising a camera and a light sourcein accordance with certain embodiments described herein.

FIG. 51 schematically illustrates an example camera comprising motorsand gears mounted to adjust pan, tilt, and focus in accordance withcertain embodiments described herein.

FIG. 52 schematically illustrates an example remote control devicecomprising a plurality of selector mechanisms (e.g., switches andbuttons) in accordance with certain embodiments described herein.

FIGS. 53A and 53B schematically illustrate example remote controldevices comprising a plurality of selector mechanisms (e.g., a rockerswitch and a plurality of buttons) configured to be operated by hand andby foot, respectively, in accordance with certain embodiments describedherein.

FIGS. 54A and 54B schematically illustrate an example remote controldevice configured to be operated by hand in accordance with certainembodiments described herein.

FIG. 55 schematically illustrates an example surgical visualizationsystem utilizing a visualization system controller in accordance withcertain embodiments described herein.

FIG. 56 schematically illustrates an example surgical visualizationsystem utilizing a visualization system controller comprising an imageanalyzer in accordance with certain embodiments described herein.

FIG. 57 is a flowchart of an example process for switching the imageacquisition subsystem selected to be the video source for the displayedimage in accordance with certain embodiments described herein.

FIG. 58 is a flowchart of an example process for analyzing the image tobe displayed and adjusting the brightness of the image to be displayedin accordance with certain embodiments described herein.

FIG. 59 is a flowchart of an example process for analyzing the image tobe displayed for peak emission response due to a duty cycle on anon-visible light source in accordance with certain embodimentsdescribed herein.

DETAILED DESCRIPTION

The following description is directed to certain embodiments for thepurposes of describing the innovative aspects of this disclosure.However, a person having ordinary skill in the art will readilyrecognize that the teachings herein can be applied in a multitude ofdifferent ways. The described embodiments may be implemented in anydevice or system that can be configured to provide visualization of asurgical site. Thus, the teachings are not intended to be limited to theembodiments depicted solely in the figures and described herein, butinstead have wide applicability as will be readily apparent to onehaving ordinary skill in the art.

Surgical Visualization System

To provide improved visualization of a surgical site, a surgical devicecan be provided with multiple integrated cameras. Each of the camerasmay capture a distinct view of the surgical site. In some embodiments,imagery from the plurality of cameras may be displayed to facilitateoperation in a surgical site. Tiled, individual, and/or stitched imageryfrom the multiple cameras can provide the user with a view of thesurgical site. The user can select the imagery to be displayed and themanner in which it is displayed for enhanced utility during surgery. Asused herein, the term imagery and images includes video and/or imagescaptured from one or more video cameras. Images from video are oftenreferred to as video images or simply images. The term images may alsorefer to still images or snap shots. Video feed or video stream may alsobe used to describe the video images such as video images from a camera.

The video cameras may comprise, for example, CCD or CMOS sensor arraysor other types of detector arrays. A frame grabber may be configured tocapture data from the cameras. For example, the frame grabber may be aMatrox Solios eA/XA, 4 input analog frame grabber board. Imageprocessing of the captured video may be undertaken. Such imageprocessing can be performed by, for example, the Matrox Supersight E2with Matrox Supersight SHB-5520 with two Intel Six Core Xeon E5645 2.4GHz processors with DDR3-1333SDRAM. This system can be designed tosupport eight or more camera inputs using two Matrox Solios eA/XA, 4input, analog frame grabber boards. More or fewer cameras may beemployed. In some implementations, a field programmable gate array(“FPGA”) can be used to capture and/or process video received from thecameras. For example, the image processing can be performed by Xilinxseries 7 FPGA boards. Other hardware devices can be used as well,including ASIC, DSP, computer processors, a graphics board, and thelike. The hardware devices can be standalone devices or they can beexpansion cards integrated into a computing system through a localcomputer bus, e.g., a PCI card or PCIe card.

FIG. 1 shows an example embodiment of a surgical visualization system 1.As illustrated, the system 1 includes a console and electronics 3 fromwhich three arms 5, 7 and 7 b extend. The first arm 7 has mounted to itsdistal end a viewing platform 9. The viewing platform may include twooculars 11 and be configured similarly to a standard surgical microscopeviewing platform. In some embodiments, however, unlike a conventionalsurgical microscope or a head mounted display the viewing platform 9 isnot a direct view device where the surgeon or other user sees directlythrough the platform, e.g., an aperture in the platform. In someembodiments, regardless whether the user can view directly through theviewing platform, the surgical visualization system 1 can be configuredto display video in a manner that the video displayed is decoupled frommovement of the surgical microscope cameras such that a user can adjustthe position and/or orientation of the surgical microscope cameraswithout moving the oculars 11 or the user adjusting position. Asdiscussed in more detail below, the viewing platform 9 may includedisplays that receive signals from cameras that the surgeon or useremploys to view the surgical site.

In some embodiments, cameras can be mounted to the viewing platform 9and the cameras can be configured to provide imagery of the surgicalsite. Accordingly, the cameras can be used to provide imagery similar toa conventional surgical microscope. For example, the cameras on theviewing platform 9 can be configured to provide a working distance, or adistance from the viewing platform 9 to the patient, that can vary usingzooming. The virtual working distance can vary, where the workingdistance can be at least about 150 mm and/or less than or equal to about450 mm, at least about 200 mm and/or less than or equal to about 400 mm,or at least about 250 mm and/or less than or equal to about 350 mm. Theworking distance can be selected and/or changed by the surgeon. In someembodiments, changing the working distance does not affect the positionand/or orientation of the oculars 11 with respect to the user orsurgeon. In various embodiments, different objectives having differentwork distances can be employed for different procedures. One objectivecan be switched out for another objective to provide a different workdistance for a different procedure. In some embodiments, zoom lenssystems are included to provide the ability to vary working distance. Insome embodiments, the cameras mounted on the viewing platform 9 can beused to provide gesture recognition to allow a surgeon to virtuallyinteract with imagery provided by the display using the surgeon's hands,a surgical tool, or both, as described in greater detail herein.

The second arm 5 has mounted to its distal end an input and displaydevice 13. In some embodiments, the input and display device 13comprises a touchscreen display having various menu and control optionsavailable to a user. In some embodiments, the touchscreen can beconfigured to receive multi-touch input from ten fingers simultaneously,allowing for a user to interact with virtual objects on the display. Forexample, an operator may use the input device 13 to adjust variousaspects of the displayed image. In various embodiments, the surgeondisplay incorporating a video camera providing a surgical microscopeview may be mounted on a free standing arm, from the ceiling, on a post,or the like. The flat panel display touch screen 13 may be positioned ona tilt/rotate device on top of the electronics console.

A surgical tool 17 can be connected to the console 3 by electrical cable19. The surgical tool 17 includes, for example, a cutting tool, acleaning tool, a device used to cut patients, or other such devices. Inother embodiments, the surgical tool 17 may be in wireless communicationwith the console 3, for example via WiFi (e.g., IEEE 802.11a/b/g/n),Bluetooth, NFC, WiGig (e.g., IEEE 802.11ad), etc. The surgical tool 17may include one or more cameras configured to provide imagery, e.g.,image and/or video data. In various embodiments, video data can betransmitted to a video switcher, camera control unit (CCU), videoprocessor, or image processing module positioned, for example, withinthe console 3. The video switching module may then output a displayvideo to the viewing platform 9. The operator may then view thedisplayed video through the oculars 11 of the viewing platform 9. Insome embodiments, the binoculars permit 3D viewing of the displayedvideo. As discussed in more detail below, the displayed video viewedthrough the viewing platform 9 may comprise a composite video formed(e.g., stitched or tiled) from two or more of the cameras on thesurgical tool 17.

In use, an operator may use the surgical tool 17 to perform open and/orminimally invasive surgery. The operator may view the surgical site byvirtue of the displayed video in the viewing platform 9. Accordingly,the viewing platform (surgeon display system) 9 may be used in a mannersimilar to a standard surgical microscope although, as discussed above,the viewing platform 9 need not be a direct view device wherein the usersees directly through the platform 9 to the surgical site via an opticalpath from the ocular through an aperture at the bottom of the viewingplatform 9. Rather, in various embodiments, the viewing platform 9includes a plurality of displays, such as liquid crystal or lightemitting diode displays (e.g., LCD, AMLCD, LED, OLED, etc.) that form animage visible to the user by peering into the ocular. Accordingly, onedifference, however, is that the viewing platform 9 itself need notnecessarily include a microscope objective or a detector or otherimage-capturing mechanisms. Rather, the image data can be acquired viathe cameras of the surgical tool 17. The image data can then beprocessed by a camera control unit, video processor, video switcher orimage processor within the console 3 and displayed imagery may then beviewable by the operator at the viewing platform 9 via the displaydevices, e.g., liquid crystal or LED displays, contained therein. Insome embodiments, the viewing platform 9 can provide a view similar to astandard surgical microscope using cameras and displays and can be usedin addition to or in conjunction with a standard surgical microscopeoptical pathway in the viewing platform. In certain embodiments, theviewing platform 9 can provide a surgical microscope view whereinchanges in the viewing angle, viewing distance, work distance, zoomsetting, focal setting, or the like is decoupled from movement of theviewing platform 9. In certain embodiments, changes in the position,pitch, yaw, and/or roll of the imaging system 18 are decoupled from theviewing platform 9 such that the imaging system 18 can move and/orre-orient while the surgeon can remain stationary while viewing videothrough the oculars 11.

The third arm 7 b can include an imaging system 18 that can beconfigured to provide video similar to a direct-view surgery microscope.The imaging system 18 can be configured, then, to provide a surgicalimaging system configured to provide an electronic microscope-like viewthat can comprise video of the work site or operational site from aposition above the site (e.g., about 15-45 cm above the surgical site)or from another desired angle. By decoupling the imagers 18 from thedisplay, the surgeon can manipulate the surgical imaging system toprovide a desired or selected viewpoint without having to adjust theviewing oculars. This can advantageously provide an increased level ofcomfort, capability, and consistency to the surgeon compared totraditional direct-view operating microscope systems. In someembodiments, as described herein, the imagers 18 can be located on theviewing platform 9, on a dedicated arm 7 b, on a display arm 5, on aseparate post, a separate stand, supported from an overhead structure,supported from the ceiling or wall, or detached from other systems. Theimagers 18 can comprise a camera configured to be adjustable to providevarying levels of magnification, viewing angles, monocular or stereoimagery, convergence angles, working distance, or any combination ofthese.

The viewing platform 9 can be equipped with wide field-of-view oculars11 that are adjustable for refractive error and presbyopia. In someembodiments, the oculars 11, or eyepieces, may additionally includepolarizers in order to provide for stereoscopic vision. The viewingplatform 9 can be supported by the arm 7 or 7 b, such that it may bepositioned for the user to comfortably view the display 13 through theoculars 11 while in position to perform surgery. For example, the usercan pivot and move the arm 7 or 7 b to re-orient and/or re-position theviewing platform 9. In some embodiments, the viewing platform 9 can bepositioned above the patient while in use by the surgeon or other user.The surgeon can then be positioned next to the patient while performingsurgery.

In some embodiments, the image processing system and the display systemare configured to display imagery placed roughly at infinity to reduceor eliminate accommodation and/or convergence when viewing the display.For example, the display system can be configured with sufficient eyerelief for the user to reduce fatigue associated with using the displaysystem. A display optical system can include one or more lenses and oneor more redirection elements (e.g., mirrors, prisms) and can beconfigured to provide light from the display that can be imaged by abinocular viewing assembly comprising a pair of oculars, objectives,and/or turning prisms or mirrors. The display devices such as liquidcrystal displays can be imaged with the objective and the pair ofoculars and display optical system within the viewing platform 9. Thebinocular assembly and display optical system can be configured toproduce an image of the displays at infinity. Such arrangements maypotentially reduce the amount of accommodation by the surgeon. Theoculars can also have adjustments (e.g., of focus or power) to addressmyopia or hyperopia of the surgeon. Accordingly, the surgeon or otherusers may view the displays through the oculars without wearing glasseseven if ordinarily prescription glasses were worn for other activities.

In certain implementations, the display optical system does not includea pair of oculars but instead includes two or more chambers that areoptically separate to form left and right eye paths. In suchimplementations, the display optical system can be baffled to preventlight communication between the left and right eye channels. To adjustfor different accommodations, the displays within the display system canbe configured to move toward and/or away from the viewer along theoptical path. This can have an effect similar to varying focal lengthsof lenses in an ocular system. In some embodiments, the display housingwith the electronic displays can change to move the displays closer orfurther from the viewer along the optical path. In some embodiments,both the display housing and the electronic displays are configured tobe adjustable along the optical path to adjust for accommodation.

In some embodiments, the viewing platform 9 can include one or moreimagers configured to provide electronic microscope-like imagingcapabilities. FIG. 2 illustrates an example surgical imaging system 51attached to an arm 7, the system 51 including one or more cameras 18mounted on a viewing platform 9. The cameras 18 can be configured toprovide imagery of a worksite. The image data can be presented on adisplay that the user can view using oculars 11 mounted on the viewingplatform 9. This design can be used to mimic other direct-viewmicroscopes, but it can also be configured to provide additionalcapabilities. For example, the surgical imaging system 51 can beconfigured to have a variable working distance without adjusting theviewing platform 9 or the articulating arm 7. In some embodiments, theworking distance can be adjusted by adjusting one or more elements ofimaging optics of the camera(s) 18 mounted on the viewing platform 9. Incertain implementations, the working distance can be adjusted byadjusting a zoom or focal length of a zoom optical apparatus. Thesurgical imaging system 51 can be configured to provide image processingcapabilities such as electronic zooming and/or magnification, imagerotation, image enhancement, stereoscopic imagery, and the like.Furthermore, the imagery from the cameras 18 can be combined withimagery from cameras on the surgical device 17. In some embodiments, thesurgical imaging system 51 can provide fluorescence images.

Although the discussion considers images from surgical tools, numerousembodiments may involve at least one auxiliary video camera 18 and oneor more other cameras that are not disposed on surgical tools but aredisposed on other medical devices. These medical devices may includedevices introduced into the body such as endoscopes, laparoscopes,arthroscopes, etc.

Accordingly, one or more displays such as the at least one display 13included in the viewing platform 9 may be used to provide a surgicalmicroscope view using one or more cameras such as the auxiliary videocamera(s) 18 as well as to display views from one or more cameraslocated on such medical devices other than surgical tools. In someembodiments, cameras from a variety of sources, e.g., surgical tools andother medical devices, in any combination, may be viewed on thedisplay(s) on the surgical platform together with the surgicalmicroscope view from the auxiliary video cameras 18. As describedherein, the displays may provide 3D thus any of the images and graphicsmay be provided in 3D.

In various embodiments, a virtual touchscreen may be provided by theauxiliary video cameras 18 or other virtual touchscreen cameras mountedto the viewing platform 9. Accordingly, in some embodiments a user mayprovide a gesture in the field of view of the auxiliary video camerasand/or virtual touchscreen cameras and the processing module can beconfigured to recognize the gesture as an input. Although the virtualdisplay has been described in the context of the auxiliary video cameras18, other cameras, e.g., virtual reality input cameras, possibly inaddition to the auxiliary video cameras 18 may be used. These camerasmay be disposed on the viewing platform 9 or elsewhere, such as thethird arm 7 b. As described herein the displays may provide 3D thus thevirtual reality interface may appear in 3D. This may increase theimmersive quality of the viewing experience, enhancing the detail and/orrealistic presentation of video information on the display.

In some embodiments, as illustrated in FIG. 3A, the surgical imagingsystem 51 includes an isocenter positioning system 52 attached to theviewing platform 9. The isocenter positioning system 52 can include asingle track or guide configured to move and orient the cameras 18 suchthat they are substantially pointed at a single point 53, the isocenter.In some embodiments, a second track or guide can be attached to thefirst guide in an orthogonal manner to provide movement along twodimensions while substantially maintaining the pointing angle towardsthe isocenter 53. Other configurations can be used to provide isocenterpointing capabilities, such as articulating arms, electro-mechanicalelements, curved friction plates, etc. In some embodiments, asillustrated in FIG. 3B, the imaging system is configured to move in anisocenter manner. This can be used to enhance dexterity of the user ofthe system because hand-eye coordination is increased or maximized. Suchenhanced dexterity can be vital for prolonged and/or difficult surgery.In the displayed embodiment, the horizons of the acquisition systems areconfigured to be horizontal to match the horizon of the display systemand the user. As shown in FIG. 3B, in various embodiments, a stereoimaging system may be maintained in a horizontal configuration as it ismoved across a range of locations to avoid confusion for the userviewing the video from the stereo camera. By maintaining a commonrelative horizon between the display and the acquisition system, theuser can relatively easily translate hand motion to manipulation ofobjects in the display, which may not be the case where translation ofthe acquisition system is accompanied by a relative rotation between thedisplay and the acquisition system.

In the embodiments illustrated in FIGS. 3A and 3B, the isocenterassemblies can be a part of the display system or a separate,independent system. For example, the viewing platform 9 can be mountedon a separate arm from the cameras 18. Thus, the display and the imageacquisition of the surgical imaging system can be decoupled, similar tothe embodiment illustrated in FIG. 1. By decoupling the isocentercameras 18 from the display ergonomic benefits are provided such as, forexample, the surgeon does not need to be looking through binoculars foran extended period of time or at an uncomfortable position or angle. Invarious embodiments, a common relative horizon for both the display andthe acquisition system may also be employed.

In some embodiments, the distance between the surgical site of interestand the imagers, e.g., the working distance, can be at least about 20 cmand/or less than or equal to about 450 cm, at least about 10 cm and/orless than or equal to about 50 cm, or at least about 5 cm and/or lessthan or equal to about 1 m, although values outside this range arepossible.

The user can interact with the surgical imaging system 51 to select aworking distance, which can be fixed throughout the procedure or whichcan be adjusted at any point in time. Changing the working distance canbe accomplished using elements on a user interface, such as a graphicaluser interface, or using physical elements such as rotatable rings,knobs, pedals, levers, buttons, etc. In some embodiments, the workingdistance is selected by the system based at least in part on the cablesand/or tubing being used in the surgical visualization system. Forexample, the cables and/or tubing can include an RFID chip or an EEPROMor other memory storage that is configured to communicate information tothe surgical imaging system 51 about the kind of procedure to beperformed. For an ENT/Head/Neck procedure, the typical working distancecan be set to about 40 cm. In some embodiments, the user's pastpreferences are remembered and used, at least in part, to select aworking distance.

In some embodiments, the working distance can be changed by translatingan imaging lens or imaging lenses along a longitudinal axis (e.g., az-axis parallel to gravity). The imaging lens(es) can be translated inan orthogonal and/or transverse direction (e.g., x- and/or y-axis) toadjust a convergence angle with changes in the working distance. In thisway, the viewing platform 9 and/or the position of the cameras 18 canremain relatively fixed with changes in working distance. This can alsoprovide for stereoscopic image acquisition, thereby providing 3D videoto the surgeon while being able to change working distance.

In some embodiments, gross focus adjustment can be accomplished manuallyby positioning the cameras 18 and arm 7. The fine focus adjustment canbe done using other physical elements, such as a fine focusing ring, orit can be accomplished electronically.

In some embodiments, the magnification of the surgical imaging system 51can be selected by the user using physical or virtual user interfaceelements. The magnification can change and can range between about 1×and about 6×, between about 1× and about 4×, or between about 1× andabout 2.5×. Embodiments may be able to change between any of these suchas between 2.5× and 6× or between 2.5× and 6×. Values outside theseranges are also possible. For example, the system 51 can be configuredto provide magnification and demagnification and image inversion, with arange from about −2× to about 10×, from about −2× to about 8×, fromabout −2× to about 4×, from about −0.5× to about 4×, or from about −0.5×to about 10×. The surgical imaging system 51 can be configured todecouple zoom features and focus adjustments, to overcome problems withtraditional operating room microscopes. In some embodiments, thesurgical visualization system 51 can be used to provide surgicalmicroscope views. In some embodiments, the surgical imaging system 51can decouple instrument myopia by providing an electronic displayinstead of a direct view of a scene. The electronic displays can beconfigured to be focused at varying levels of magnification allowing theuser to view the displays without adjusting the oculars betweenmagnification adjustments. Moreover, in various embodiments, the ocularscan be configured to provide continuous views at infinity. In someembodiments, however, the principal user of the surgical imaging systemmay select an accommodation level for the oculars, rather than using arelaxed view provided by the electronic displays. The electronicdisplays, in various embodiments, however, can remain in focus and theocular adjustments do not affect the focus of the various videoacquisition systems. Thus, adjustments by the principal user do notaffect the views of the other users of the system viewing, for example,other displays showing the video, as the cameras/acquisition systems canremain focused. In some embodiments, the surgical imaging system 51 canbe focused at a relatively close working distance (e.g., a distance witha relatively narrow depth of field) such that the image remains focusedwhen moving to larger working distances (e.g., distances with broaderdepth of field). Thus, the surgical imaging system 51 can be focusedover an entire working range, reducing or eliminating the need torefocus the system after magnification or zoom adjustments are made.

In some embodiments, the surgical imaging system 51 includes an afocalzoom assembly to provide changes in magnification. The afocal zoomassembly comprises an afocal system that can be used to providevariations in magnification at a fixed working distance and/or toprovide variations in magnification with changes in working distance. Incertain embodiments, the surgical imaging system 51 includes a commonobjective for left and right optical paths (e.g., left and right lenssystems) corresponding to left and right imaging sensors configured toproduce images for the left and right eyes of the user. The left andright lens systems can each comprise one or more lenses or lens groupsdisposed in each of the respective left and right paths. The commonobjective (e.g., a single lens, compound lens, or lens group) can beconfigured to have a focal length corresponding to a distance to theobject. The afocal zoom assembly can thus receive collimated light fromthe objective and produce collimated light while changing amagnification of the optical system of the surgical imaging system 51.The zoom or variable magnification can provide selectable magnificationor zoom. In certain embodiments, the surgical imaging system 51 includesa variable diaphragm to adjust the aperture of the optical system. Thevariable diaphragm can be adjusted to increase or decrease the f-numberof the optical system or the variable diaphragm can be adjusted tomaintain a relatively constant f-number with changes in themagnification of the optical system.

In some embodiments, the surgical imaging system 51 includes a zoomoptical system that functions to change working distance and/ormagnification. The surgical imaging system 51 can also include a zoomobjective lens and one or more afocal zoom assemblies.

The surgical imaging system 51 can include a gimbal system to point thecamera(s), for example, objective as desired. In various embodiments,the gimbal system is configured to provide isocentric views of theworksite or surgical site. The gimbal system can be configured to changethe relative position and/or orientation of imaging optics to maintain aisocentric views. In some embodiments, the gimbal system can beconfigured to move the imaging system (e.g., the optical assembly) whilemaintaining the isocentric view while also maintaining the positionand/or orientation of the oculars of the surgical imaging system 51relatively motionless.

FIGS. 4A and 4B illustrate an embodiment of the surgical imaging system51 having an optical system 53 mounted under the viewing platform 9. Asillustrated, the optical components are shown as free-standing to showthe structure of the components, but in practice the optical components53 will be mounted within or on a structure attached to the viewingplatform. In some embodiments, the optical system 53 and/or the cameras18 (discussed above) can be modular and can be selected and swapped foruse with the surgical imaging system 51.

The optical system 53 is configured to provide stereo image data to theimaging system 51. The optical system 53 includes a turning prism 54 tofold the optical path underneath the viewing platform 9 to decrease thephysical extent (e.g., length) of the imaging system under the viewingplatform 9. The turning prism 54 can be configured to fold the opticalpath from vertical to horizontal. This design can reduce the thickness(e.g., vertical dimension) of the optical system 53. This configurationcan also reduce the size of any housing used to house the optical system53. This approach can also reduce the size of the viewing platform 9,for example, where the optical system is incorporated into the housingof the viewing platform 9.

The optical system 53 can include, in some embodiments, an objectivelens or objective lens group, an afocal zoom lens group, an imaging lensor lens group, and an image sensor or other similar detector. The afocalzoom lens group can be adjusted or manipulated to receive and produce acollimated beam wherein the adjustments alter the zoom or magnificationof the optical system 53. The optical system 53 can include a commonobjective lens or objective lens group for both right and left opticalpaths (e.g., corresponding to right and left eye views for astereoscopic display). Alternatively, the optical system 53 can includea separate objective lenses or lens groups for both right and leftoptical paths (e.g., corresponding to right and left eye views for astereoscopic display). In various embodiments, the objective provides acollimated beam to the afocal zoom. The afocal zoom may also output acollimated beam in certain embodiments. In some embodiments, the opticalsystem 53 is configured to translate the imaging lens or imaging lensgroup along a longitudinal axis or vertical axis (e.g., a z-axis) tochange the working distance. The optical system 53 can also beconfigured to translate the imaging lens or imaging lens group along atransverse or horizontal axis (e.g., a x- and/or y-axis) to alter theconvergence angle. In some implementations, alterations of theconvergence angle correspond to changes in the working distance tomaintain appropriate convergence at a targeted location or distance. Forexample, as the working distance decreases, the convergence angle can bemade to increase. In certain embodiments, the convergence angle of theoptical system 53 can be about 3 degrees at about 300 mm workingdistance, or between about 2 degrees and about 5 degrees at betweenabout 150 mm and about 500 mm working distance, or between about 1degree and about 10 degrees at between about 50 mm and about 1000 mmworking distance. Values outside these ranges are also possible.

In some embodiments, the optical system 53 includes a zoom opticalsystem that functions to change working distance and/or magnification.The optical system 53 can also include a zoom objective lens and one ormore afocal zoom assemblies.

In some embodiments, the optical system 53 comprises a Greenough-stylesystem wherein the optical paths for each eye have separate opticalcomponents. In some embodiments, the optical system 53 comprises aGalilean-style system wherein the optical paths for each eye passthrough a common objective. The Greenough-style system may be preferablewhere imaging sensors are being used to capture and convey the imagedata as compared to the Galilean-style system. The Galilean system canintroduce aberrations into the imagery by virtue of the rays for eacheye's optical path passing through a periphery of the objective lens.This does not happen in the Greenough-style system as each optical pathhas its own optics. In addition, the Galilean system can be moreexpensive as the objective used can be relatively expensive based atleast in part on the desired optical quality of the lens and its size.

The optical system 53 can include two right-angle prisms 54, two zoomsystems 55, and two image sensors 56. This folding is different from atraditional operating room microscope because the optical path leads toimage sensors rather than to a direct-view optical system.

In some embodiments, the optical system 53 can have a relativelyconstant F-number. This can be accomplished, for example, by varying thefocal length and/or aperture of the system based on working distanceand/or magnification. In one embodiment, as the focal length changes,the eye paths can move laterally apart (or together), the prisms 54 canrotate to provide an appropriate convergence angle, and the aperturescan change their diameters to maintain the ratio of the focal length tothe diameter a relatively constant value. This can produce a relativelyconstant brightness at the image sensor 56, which can result in arelatively constant brightness being displayed to the user. This can beadvantageous in systems, such as the surgical visualization systemsdescribed herein, where multiple cameras are being used and changing anillumination to compensate for changes in focal length, magnification,working distance, and/or aperture can adversely affect imagery acquiredwith other cameras in the system. In some embodiments, the illuminationcan change to compensate for changes in the focal length and/or theaperture so as to provide a relatively constant brightness at the imagesensors 56. In some embodiments, illumination can be provided throughthe objective lens or objective lens group of the optical assembly 53.

The optical assembly 53 can include a zoom system 55 configured toprovide a variable focal distance and/or zoom capabilities. AGalilean-style stereoscopic system generally includes a common objectivefor the two eye paths. When this optical system is imaged with imagesensors 56, it can create aberrations, wedge effects, etc. that can bedifficult to compensate for. In some embodiments, the surgical imagingsystem 51 can include a Galilean-style optical system configured tore-center at least one of the stereo paths to a central location throughthe objective lens, which can be advantageous in some applications.

In some embodiments, the real-time visualization system utilizes aGreenough-style system. This can have separate optical components foreach stereo path. The optical assembly 53 can be configured to providevariable magnification and/or afocal zoom and can be configured tooperate in a magnification range from about 1× to about 6×, or fromabout 1× to about 4×, or from about 1× to about 2.5×.

The distal-most portion of the Greenough assembly 53 can be similar infunctionality to an objective lens of a typical, direct-view operatingroom microscope with the working distance set approximately to that ofthe focal length. The working distance, and in some implementations thefocal length, can be between about 20 cm and about 40 cm, for example.In some embodiments the work distance may be adjustable from 15 cm to 40cm or to 45 cm. Other values outside these ranges are also possible. Insome embodiments, the surgical imaging system 51 includes anopto-mechanical focus element configured to vary the focal length of apart of the optical assembly 53 or the whole optical assembly 53.

FIG. 5A illustrates an example embodiment of an optical assembly 53 foruse in a stereoscopic surgical imaging system, such as those describedherein with reference to FIGS. 4A-4B. FIG. 5A illustrates a side view ofan example optical assembly 53 configured to use a turning prism 54 tofold an optical path from a tissue 57 to a sensor 56 along a lens train55 that is situated near or adjacent to a viewing platform 9. This canadvantageously provide a relatively long optical path in a relativelycompact distance.

In some embodiments, the optical assembly 53 includes a source ofillumination, such as a fiber optic or light emitting diode (LED) orother type of light source providing light through one or more of theoptical elements of the optical assembly 53. In certain implementations,the light source (e.g., fiber optic, LED, etc.) provides illuminationthrough the objective lens or objective lens group to provideillumination to the worksite. (A beamsplitter or beam combiner may beused to couple light into the optical path or the light source mayitself be disposed in the optical path.) This arrangement can be usefulas the light from the light source (e.g., fiber optic, LED, etc.) can bedirected to the worksite along a similar optical path as light arrivingfrom the worksite. This configuration can help the user to control thelocation and/or amount of illumination provided by the light source(e.g. fiber optic, LED, etc.). For example, the user can direct theillumination to the same location that the user is viewing. As anotherexample, changes in the relative positions of the objective lens andlight source (e.g., fiber optic, LED, etc.) can change the divergence ofthe light to increase or decrease the amount of light per unit area atthe worksite. As another example, changes in the focal length of theobjective lens group can change the divergence of the light from thelight source (e.g., fiber optic, LED, etc.) to control illumination atthe worksite. In some embodiments, additional optics such as beamshaping optics may be included for the light source. One or more lensesmay for example be disposed forward of the fiber optic or LED or otherlight source and may be adjustable, to control divergence and/or beamsize.

In some embodiments, the source of illumination can be controlled usinga gimbal. The light source may be mounted on a gimbal or one or moreother translation/rotation devices to provide the ability tocontrollably redirect the beam into different directions. The gimbal orother orientation control device may be moved manually or haveelectrically driven actuators to control movement of the system and thedirection of the beam. The gimbal can be used to provide illuminationwith variable pitch. The gimbal can also be used to steer theillumination to provide light at a targeted location, with a targetedintensity, and/or from a desired or selected angle.

Multiple light sources may be employed. These multiple light sourcesmay, for example, be on opposite sides of the optical path. In someembodiments, different light sources are mounted on different gimbals(or motion and/or orientation control systems) such as described aboveand elsewhere herein. Accordingly, a plurality of beams may becontrolled so that a first beam from a first light source may bedirected in a first direction (possibly using a first gimbal system) anda second beam from a second light source may be directed in a seconddifferent direction (possibly using a second gimbal system) and thefirst and second light sources and directions can be separately changedsubsequently. In some embodiments, the size of the beam at the object isless than the field of view of the camera imaging the object. Themultiple light sources can be used to fill a larger portion of thatfield of view that is larger than the beam from a single light source.In various embodiments, the plurality of beams from the plurality oflight sources fills at least as large as the field of view of the one ormore cameras imaging the object.

In various embodiments, as the optical assembly changes zoom ormagnification, the illumination from the light source (e.g., fiber opticor LED) can change. For example, variable divergence of the illuminationcan accompany changes in zoom of the optical assembly 53 to place morelight energy in an area for imaging that area. In some implementations,the fiber optic illumination source can be adjusted to maintain arelatively constant divergence with changes in zoom of the opticalassembly 53 by placing the illumination at a place in the opticalassembly 53 or elsewhere where the optical properties remain relativelyconstant with changes in zoom.

In some embodiments, the optical assembly 53 can include a commonobjective lens with one or more zoom lenses (e.g. Galileanconfiguration) or can have separate objectives for separate left andright optical paths (e.g., Greenough configurations) with the fiberoptic illumination or other illumination. This arrangement can be usedto transform the effective numerical aperture of the fiber optic (orother type of light source) to achieve targeted or desired illuminationeffects. In some embodiments, the source of illumination is remote fromthe optical assembly 53 and may be delivered to the worksite through thefiber optic. In other embodiments, however, as described above, insteadof employing a fiber optic, a light source such as a light emittingdiode (LED) or other emitter (e.g., solid state emitter), with orwithout beam shaping optics (e.g., one or more lenses) can be employedinstead of a fiber. One or more fibers and one or more light sources(such as multiple LEDs) can be employed, for example, at differentlocation such as on opposite sides of the optical path or surgical site.Similar combination of one or more fiber and one or more light sourcesuch as one or more LEDs can be used. As described above, these can beincluded in the optical system in some embodiments such that the lightfrom the fiber or light source propagates through the camera optics usedfor imaging (e.g., objective, afocal zoom, and/or imaging lens) toilluminate the surgical site or a portion thereof. In variousembodiments, the light from the light source propagates through theobjective (and not the afocal zoom and/or imaging lens) prior to beingincident on the object. Such a configuration can reduce the amount ofback reflected from the optical surfaces that results in light incidenton the sensor. In various embodiments, a beam splitter or beam combiner(e.g., prism) may be use to couple the illumination beam into theoptical path of the camera. In other cases, the light source may bedisposed itself in the optical path. In some embodiments the lightsource (e.g., fiber, LED, etc.) is disposed adjacent to the camera anddoes not couple light such that the light propagates through the cameraoptics prior to being incident on the surgical site.

Light sources may be used in connection with providing illumination ofthe object for the one or more proximal cameras disposed above thesurgical site or body by, e.g., 25-40 mm, for one or more camerasmounted on a surgical tool, as well as for one or more cameras thatprovide a surgical microscope view, for other cameras or for anycombination thereof. The light sources and illumination configurationsincluding for example the gimbals or other positioning orientationdevices may be useful for any of these applications (e.g., one or moreproximal cameras, one or more cameras on a surgical tool(s), etc.) andmay be used for other types of cameras (stereo or otherwise) as well. Asdiscussed above, the cameras may have a Galilean or Greenough likeconfiguration and illumination may be provided through at least aportion of the camera optics (e.g., one or more camera lenses) in eitherthe left or right channels or both in the case of stereo cameras.

Movement Control System

FIGS. 6A-C illustrate embodiments of components of a movement controlsystem 10100 that can be configured to allow an operator of the surgicalvisualization system 1, such as a medical professional or assistant, tocontrol the movement of one or more imagers 18. Such imagers maycomprise cameras that provide a surgical microscope view through theoculars 11 or eyepieces of the binocular display unit 9. In variousembodiments, the movement control system can enable the imagers 18 to bemoved without changing the positioning of oculars 11, and thus anoperator can remain in an ergonomic position while changing the viewprovided by the imager 18. The imager 18 can be on the binocular displayunit 9 or located elsewhere such as on a separate platform orarticulated arm. Additionally, unlike conventional articulated opticalsystems which are generally unwieldy, complex, and have the potentialfor introducing optical aberrations, use of the movement control system10100 with the surgical visualization system 1 can result in asimplified system with greater optical clarity and range of movement. Itshould be appreciated by one of skill in the art that, while thedescription of the movement control system 10100 is described herein inthe context of medical procedures, the movement control system 10100 canbe used for other types of visualization and imaging systems. Movementof the imagers 18 can be performed prior to and/or during the activity,such as surgical procedures, dental procedures, and the like. Movementof the imagers 18 can advantageously allow a medical professional orother operator to alter the view through oculars 11, for example, toprovide different surgical microscope-like electronic visualizationswhich might be beneficial during the course of a medical procedure orfor different surgical procedures.

In some embodiments, control of the movement of the imager 18 can beachieved using a single control member such as 10110. This provides theadvantage of allowing single-handed operation of the movement controlsystem 10100 which can, for example, allow a medical professional tomove one or more imagers 18 using only one hand while using a secondhand for other tasks such as performing surgical techniques. It shouldbe appreciated by one of skill in the art that, while the description ofthe movement control system 10100 is described herein in the context ofmedical procedures, the movement control system 10100 can be used forother types of visualization and imaging systems.

Operation

As illustrated in FIGS. 6A-C, in some embodiments, the control member,such as a joystick, 10110 can be used to translate the imager 18, adjustthe pitch, yaw, and/or roll of the imager 18, and/or adjust the workingdistance of the imager 18. In some embodiments, the oculars 11 canremain immobile when translating the imager 18, adjusting the pitch,yaw, and/or roll of the imager 18, and/or adjusting the working distanceof the imager 18. The ability for a single control member 10110 tocontrol translation, adjustments to pitch and/or yaw, and/or adjustmentsto the working distance can beneficially simplify operation of thedevice as an operator need not release the control member 10110 tocontrol multiple aspects of its operation. For example, an operator cantranslate the imager 18 and subsequently adjust the pitch and/or yawwithout having to release the control member 10110 thereby increasingease-of-use of the system and enhancing efficiency when using thissystem.

As shown in FIG. 6C, one or more control members of the movement controlsystem 10100, such as control member 10110, and/or one or more imagerarms (see FIG. 7) can be attached to a component of the movement controlsystem 10100 using various types of joints and/or can be remote from themovement control system 10100 such as a remote joystick or toggle. Insome embodiments, the control member 10110 can include a joint forattachment to the movement control system 10100. For example, as shownin the illustrated embodiment, control member 10110 can include joint10111. In some embodiments, one or more of the joints can includecomponents for detecting movement of the control member and/or an imagerarm. For example, one or more of the joints can include one or moresensors for detecting rotation and/or translation of the control memberand/or the imager arm about the joint. The signals from these sensorscan be used to control other components of the movement control system,such as one or more electromechanical components.

For purposes of this disclosure, rotation about joints, such as joint10111, around the x-axis is hereinafter termed “pitch” or “tilt” androtation about joints, such as joint 10111, around the y-axis ishereinafter termed “yaw” or “pan.”

As shown in the illustrated embodiment, the joint 10111 can be sphericaljoints received in a socket formed in the member 10220 thereby forming aball-and-socket attachment. As should be apparent to one of ordinaryskill in the art, other types of mounting mechanisms may be used forattaching control member 10110 as well as an imager arm to components ofthe movement control system 10100. For example, joints such as gimbalscan be used which limit the rotational degrees of freedom about thegimbal. Other types of joint can be used depending on the types ofmovement the movement control system is designed to allow. For example,if only pitch is needed without yaw, one can use a joint having a singlerotational degree of freedom. In some embodiments, the control member10110 can be positioned remotely from the movement control system 10100.

General Embodiment

With continued reference to FIGS. 6A and 6B, in some embodiments, themovement control system 10100 can be attached to an attachmentstructure, such as binocular display unit 9, and support one or moreimagers 18. As shown in the illustrated embodiment, the movement controlsystem 10100 can be oriented generally underneath the binocular displayunit 9 and in some embodiments can be sized such that the movementcontrol system 10100 does not extend significantly beyond the outerhousing of the binocular display unit 9. This can advantageously providea smaller form factor thereby reducing the likelihood that the movementcontrol system 10100 will interfere with the medical professionals andassistants during a medical procedure. In other embodiments, theattachment structure can be other components of the surgicalvisualization system 1 such as, but not limited to, a dedicatedarticulating arm or a display arm. In some embodiments, the movementcontrol system 10100 can extend significantly beyond the outer housingof the binocular display unit 9 or any other platform to which it isattached. This can be advantageous in situations where a greater degreeof movement of the imagers 18 is desired or in embodiments where thecontrol member 10110 is located above the attachment point between themovement control system 10100 and binocular display unit 9.

With continued reference to FIGS. 6A and 6B, as discussed in part above,the movement control system 10100 can be configured to allow translationof one or more attached imagers 18 along a plane relative to thebinocular display unit 9. In some embodiments, the binocular displayunit 9 can be immobile while the one or more imagers 18 are translated.For example, when attached to the binocular display unit 9 with themovement control mechanism 10100 parallel to an operating table 10101,the one or more imagers 18 can be translated along a plane parallel tothe operating table 10101. As shown in the illustrated embodiment, themovement control system 10100 can be translated along both the x-axisand the y-axis (which projects perpendicularly through the sheet). Thiscan advantageously allow the medical professional to position the viewof oculars 11 for comfortable viewing by the surgeon thereby reducingphysical strain on the surgeon during long procedures.

In some embodiments, defining an imager 18 centered on the movementcontrol system 10100 (as shown in FIG. 6A) as having an x-axis, y-axis,and z-axis coordinate of zero, the movement control system 10100 canhave a range of translation relative to the binocular display unit 9, ofapproximately ±500 mm along the x-axis and y-axis at full extension,approximately ±400 mm along the x-axis and y-axis at full extension,approximately ±300 mm along the x-axis and y-axis at full extension,approximately +200 mm along the x-axis and y-axis at full extension, orapproximately ±100 mm along the x-axis and y-axis at full extension. Insome embodiments, full extension along one axis can be greater than fullextension along the other axis. For example, in some embodiments, fullextension along the x-axis may be approximately ±175 mm whereas they-axis extension can be three-quarters full extension of the x-axis,one-half full extension of the x-axis, one-quarter full extension of thex-axis, or any other ratio between unity and zero. In some embodiments,the range of translation relative to the binocular display unit 9 alongthe y-axis can be approximately ±87.5 mm. This can be advantageous incases where allowing the y-axis to have a full range of motion mayinterfere with the medical professional and/or assistants.

These ratios can be reversed such that the range of translation of thex-axis can be three-quarters full extension of the y-axis, one-half fullextension of the y-axis, one-quarter full extension of the y-axis, orany ratio between unity and zero. Additionally, in some embodiments, theimager 18 can translate further in the “positive” direction than the“negative” direction. For example, along the x-axis, the imager 18 maymove from −100 mm to 500 mm. Ranges of motion outside these ranges arealso possible. As should be apparent to one of ordinary skill in theart, the maximum translation relative to the binocular display unit 9along the x-axis and y-axis can be chosen to provide a balance betweengreater maneuverability, the yaw and/or pitch angles, working distances,size constraints, and other such factors.

As described in part above and as will be discussed in greater detailbelow, in some embodiments, translation of the imagers 18 can beperformed by translating one or more control members, such as controlmember 10110, in the desired direction. In some embodiments, the controlmember 10110 can be electrically coupled to the movement control system10100 to provide translation via an electromechanical system utilizingstepper motors, linear motors, or the like. For example, a joint of thecontrol member 10110 can include components for detecting translation ofthe control member 10110. The signals from these sensors can be used tocontrol other components of the movement control system, such as one ormore electromechanical components such as stepper motors, linear motors,or the like to translate the imager 18. The electromechanical componentscan be coupled to a moveable platform to which the imager 18 can beattached. In some embodiments, the control member 10110 can bephysically connected to the movement control system 10100 without anyelectromechanical assistance.

As should be appreciated by one of ordinary skill in the art, themovement control system 10100 need not translate solely along a planeparallel to the operating table 10101 or the x-y plane as set forth inthe illustrated embodiment. In some embodiments, the plane oftranslation can be defined by the orientation of the mount to which themovement control system 10100 is connected. In some embodiments, themovement control system 10100 can be configured for non-planartranslation and/or translation along more than one plane. In someembodiments, for example, a tip and tilt stage provides angular motion.A rotary stage can also be used to provide rotary motion.

With continued reference to FIGS. 6A and 6B, as described in part above,the movement control system 10100 can be configured to allow rotation ofthe one or more attached imagers 18 about a joint which can be attachedto components of the movement control system 10100 and/or remotely fromthe movement control system 10100. In some embodiments, the movementcontrol system 10100 can be designed to allow the control member, suchas control member 10110, as well as the imager 18 and/or imager arm to“pitch” or “tilt” and “yaw” or “pan” relative to the binocular displayunit 9. In some embodiments, the binocular display unit 9 can beimmobile while the “tilt” and “yaw” or “pan” of the one or more imagers18 are adjusted. Pitch or yaw can allow the imager 18 to have a line ofsight that is centered (e.g., focused) on the surgical site after theimager 18 is translated. This can advantageously allow the medicalprofessional or assistant to adjust the viewing angle during a medicalprocedure. This can be beneficial in circumstances where a medicalprofessional is unable to adequately view an object due to anotherelement obstructing the view. Under such circumstances, a medicalprofessional can translate the imager 18 and adjust the viewing angle ofthe imager 18 such that the same general area is viewed from a differentangle.

In some embodiments, defining an imager 18 in a perpendicularorientation to the movement control system 10100 (as shown in FIG. 6A)as having an a pitch and yaw of zero (i.e., as shown in FIG. 6A), themovement control system 10100 can allow both pitch and yaw adjustmentsrelative to the binocular display unit 9 within the range ofapproximately ±60 degrees each, by approximately ±50 degrees each, byapproximately ±40 degrees each, by approximately ±30 degrees each, byapproximately ±20 degrees each, or approximately ±10 degrees each. Insome embodiments, the pitch and yaw can have different adjustmentranges. For example, in some embodiments, the yaw can have an adjustmentrange of approximately ±40 degrees whereas the pitch can have anadjustment range of approximately three-quarters that of the yaw,one-half that of the yaw, one-quarter that of the yaw, or any otherratio between unity and zero. In some embodiments, the pitch can have anadjustment range of approximately ±20 degrees.

The adjustment range of yaw and pitch can correspond to the distance atfull extension along both the x-axis and the y-axis. For example, insome embodiments, the pitch and yaw can be chosen such that the imager18 can remain centered on the surgical site when the movement controlsystem 10100 is fully extended in any direction. In some embodiments,the working distance between the imager 18 and the surgical site can beapproximately ±200 mm, with a range of translation along the x-axis ofapproximately ±175 mm, and a range of translation along the y-axis ofapproximately ±87.5 mm. In order to remain centered on the surgicalsite, the pitch adjustment range can be ±20 degrees and the yawadjustment range can be ±40 degrees. As such, because the full extensionneed not be the same in both directions, the pitch and yaw adjustmentranges can also be different to match the differences in extension. Inother embodiments, such as those in which the working distance can beadjusted, the pitch and yaw adjustment range can be chosen such that theimager 18 can remain centered on the surgical site when the movementcontrol system 10100 is fully extended in any direction at least oneworking distance. For example, in embodiments where the working distancecan be adjusted between approximately ±200 mm and 400 mm, the pitch andyaw adjustment range can be approximately ±20 degrees and approximately±10 degrees respectively to allow centering at a working distance of 400mm.

Additionally, in some embodiments, the imager 18 can adjust further in a“positive” angle than a “negative” angle. For example, the yaw may rangefrom −5 degrees to 15 degrees.

As described in part above and as will be discussed in greater detailbelow, in some embodiments, increasing or decreasing the pitch and/oryaw of the imagers 18 relative to the binocular display unit 9 can beachieved by increasing or decreasing the pitch and/or yaw of the one ormore control members, such as control member 10110. In some embodiments,the control member 10110 can be electrically coupled to the movementcontrol system 10100 to provide pitch and yaw via an electromechanicalsystem utilizing stepper motors, linear motors, or the like. Forexample, a joint of the control member 10110 can include components fordetecting pitch and/or yaw of the control member 10110. In someembodiments, the joint of the control member 10110 can be gimbals whichcan detect pitch and/or yaw of the control member 10110. The signalsfrom these sensors can be used to control other components of themovement control system, such as one or more electromechanicalcomponents such as stepper motors, linear motors, or the like to adjustthe pitch and/or yaw of the imager 18. As should be appreciated by oneof ordinary skill in the art, in some embodiments, the movement controlsystem 10100 can be configured to allow rotation along other axes suchas the z-axis. In some embodiments, the control member 10110 can bephysically connected to the movement control system 10100 without anyelectromechanical assistance.

Additionally, in some embodiments, the movement control system 10100 canbe configured to adjust the working distance between the imagers 18 andthe surgical site. In some embodiments, the binocular display unit 9 canremain immobile while the working distance of the imagers 18 isadjusted. In some embodiments, the working distance can range frombetween approximately 1 m to approximately 10 mm, from betweenapproximately 800 mm to approximately 50 mm, from between approximately600 mm to approximately 100 mm, or from between approximately 400 mm toapproximately 200 mm. In some embodiments, the control member 10110 canbe electrically coupled to the movement control system 10100 to provideworking distance adjustment via an electromechanical system utilizingstepper motors, linear motors, or the like. For example, a joint of thecontrol member 10110 can include components for detecting rotation ofthe control member 10110 about the longitudinal axis. The signals fromthese sensors can be used to control other components of the movementcontrol system, such as one or more electromechanical components such asstepper motors, linear motors, or the like to adjust the pitch and/oryaw of the imager 18. In some embodiments, the control member 10110 canbe physically connected to the movement control system 10100 without anyelectromechanical assistance.

In some embodiments, the movement control system 10100 can include atranslation system for translating an imager 18 and/or an imager arm, apitch-yaw adjustment system for adjusting the pitch and/or yaw of theimager 18 and/or an imager arm, a control member, such as control member10110, and one or more imager arms to which the imager 18 can beattached. In some embodiments, a working distance adjustment system canbe included which can allow adjustments in working distance of theimager 18 and/or an imager arm. It should be appreciated by one ofordinary skill in the art that the translation system, the pitch-yawadjustment system, and/or the working distance adjustment system can beused separately or in any combination.

Operation of the translation, pitch-yaw adjustment, and/or workingdistance adjustment systems can be performed using a control member,such as control member 10110. In some embodiments, control member 10110can be operatively coupled to the translation, pitch-yaw adjustment,and/or working distance adjustment systems. For example, as describedabove, in some embodiments, the control member can be coupled to anelectromechanical system for controlling the translation, pitch-yawadjustment, and/or working distance adjustment systems. The controlmember can be directly attached to a component of the movement controlsystem 10100 or can be remotely positioned (e.g., a toggle or joystickon a separate module). In some embodiments, the control member can becoupled directly to the translation, pitch-yaw adjustment, and/orworking distance adjustment systems such that no electromechanicaldevices are used. In some embodiments, the operator can be given theoption of controlling the translation, pitch-yaw adjustment, and/orworking distance adjustment systems with or without electromechanicaldevices. For example, the operator can control the translation,pitch-yaw adjustment, and/or working distance adjustment systems withoutelectromechanical devices for certain portions of a procedure and usesuch electromechanical devices for controlling the translation,pitch-yaw adjustment, and/or working distance adjustment systems duringother portions of a procedure. As another example, in some embodimentscoarse control of the movement control system 10100 can be achievedwithout use of electromechanical devices whereas fine control of themovement control system 10100 can be achieve with use ofelectromechanical devices, vice-versa, or a combination of the two.

In some embodiments, the movement control system 10100 can include acontrol system which controls functions of the electromechanicaldevices. In some embodiments, the electromechanical components can beprogrammed such that the electromechanical components can orient thetranslation, pitch-yaw adjustment, and/or working distance adjustmentsystems in certain positions based on the operator's input. For example,the electromechanical components can be programmed such that it goes toreverts back to a pre-set or previous position upon receiving a commandfrom the operator. As another example, the electromechanical componentscan be programmed such that an operator can specify a desired positionfor the imager 18 and the control system can control theelectromechanical devices coupled to the translation, pitch-yawadjustment, and/or working distance adjustment systems orient the imager18 in the desired position.

With reference to FIG. 7, in some embodiments, the imager arm 10120 andthe imager 18 can be attached such that the imager 18 can be directedtowards the side of the head of a patient. For example, in someembodiments, the imager 18 can be attached to the imager arm 10120 usinga yoke 10125 which can be designed to allow for coarse and/or finecontrol of pitch, yaw, and/or roll of the imager 18. In someembodiments, the yoke 10125 can have one or more pivots which can beconfigured to allow the imager 18 to have a viewing angle parallel tothe operating room floor such that an operator can view the side of thehead. In some embodiments, the yoke 10125 can be configured to allow theimager 18 to rotate such that the imager can be directed to a portion ofthe back of the head.

In some embodiments, the imager 18 can be positioned on a movementcontrol system 10130 providing at least two rotational degrees offreedom and/or at least one translational degree of freedom. In someembodiments, movement control system 10130 can provide two rotationaldegrees of freedom and at least two translation degrees of freedom. Forexample, as shown in FIG. 8, the movement control system 10130 can allowfor rotation along axis 10135 of the movement control system 10130and/or along axis 10140 (which can be parallel with the z-axis).Moreover, as shown in the illustrated embodiment, the movement controlsystem can allow translation along both the x-axis and y-axis. In someembodiments, apparatus 10130 can provide at least one translationaldegree of freedom.

As shown in the illustrated embodiment, the movement control system10130 can include a one or more control members, such as control member10145. Control member 10145 can be positioned such that the longitudinalaxis of the control member 10145 is parallel with and/or collinear withaxis 10135. This can advantageously allow the imager 18 to be rotatedabout axis 10135 by rotating the control member 10145. In someembodiments, the control member 10145 can be mechanically coupled to theimager 18. In some embodiments, the control member 10145 can be coupledto the imager 18 via an electromechanical system. For example, thecontrol member 10145 can include sensors for detecting rotation of thecontrol member 10145 and use data received from the sensors to rotatethe imager 18 via electromechanical components such as stepper motors,linear motors, or the like.

As shown in the illustrated embodiment, the movement control system10130 can include a first plate element 10150 and a second plate element10155 which can be rotatable coupled. The second plate element 10155 caninclude first and second supports 10160, 10165 to which the imager 18can be attached. In some embodiments, the first and second plateelements 10150, 10155 can be rotatable coupled such that the axis ofrotation of the two plate elements 10150, 10155 is parallel and/orcollinear with axis 10140.

In various embodiments, the gimbal advantageously allows movement of thecamera without movement of the display such as the binocular display.Such movement can include pitch or yaw, as well as possibly, x, y, or zmotion or any combination thereof. Despite such movement of the camerathat provides surgical microscope views, the binocular display need notmove similarly. Accordingly, in various embodiments a joint is providedbetween the camera and binocular display that permits pitch or yaw, aswell as possibly, x, y, or z or any combination thereof of the stereosurgical microscope view camera without requiring the same motion(include pitch or yaw, as well as possibly, x, y, or z or anycombination thereof) of the binocular display. The binocular display isthus decoupled from the camera. Such a decoupling of motion is possible,even if the camera is connected to the binocular display. For example,the camera may move laterally in the x direction and or up and down inthe y direction or may be rotated about the x or y axis to introduce yawand/or to introduce pitch, however, the binocular display (and theoculars) need not move similarly or need not move at all such that thesurgeon need not reorient his or her head to view the images on thedisplay. Movement and/or positioning and/or orientation control systems,other than gimbal systems may be employed as well. By decoupling themovement of the camera from that of the binocular display, even forcamera's mounted on or connected to the binocular display, ergonomicbenefits can be achieved. For example, the surgeon need not contorttheir neck in positions that are uncomfortable for long surgicalprocedures.

In various embodiments, the gimbal does not provide roll of thecamera(s), for example, about the axis 10140. If for example the cameracomprises a stereo camera with separate left and right cameras, suchroll would raise the left channel above the right or vice versa. Ahorizontal line through the line of sight of the left and right channelsmight not therefore be parallel with the floor (perpendicular to thegravity vector). This roll might therefore cause disorientation and/ordiscomfort for the viewer. Accordingly, various embodiments of thegimbal or other positioning/orientation system for the camera's thatprovide surgical microscope views are configured not to such roll.Substantially reducing or eliminating roll might apply to use for thecamera for surgery either or both in the downward view (for example, forspine surgery) as well as the oblique view (for temporal approach intothe skull). Such configurations that substantially reduce or eliminateroll may be applicable for gimbal systems for other cameras includingone or more proximal cameras disposed outside the surgical site adistance from the patient's body but in close proximity thereto (forexample on a stereotactic frame, etc.), such as for example, a distanceof between 5 mm and 50 mm, between about 20 mm and 40 mm (e.g., between10 mm to 25 mm) from the patient's body and/or surgical site. Aplurality of cameras including possibly a plurality of stereo camerasand possibly one or more mono-cameras (for example at 3, 6, 9, and 12o'clock positions) can be repositioned and/or reoriented usingpositioning and orientations devices potentially in x, y, and zdirections as well as in pitch and yaw and any combination thereof.However, in various embodiments such positioning and/or orientationdevices do not permit the amount of roll to exceed that which wouldcause disorientation or do not provide for roll of the left and rightchannels of the stereo cameras altogether so as to reduce disorientationfor the viewer.

In some embodiments, the control member 10145 can include one or moreswitches and/or actuators 10170 for controlling movement of the device.For example, the actuator 10170 can be coupled to mechanisms which canunlock the apparatus 10130 such that the movement control system 10130can be manipulated to rotate and/or translate the imager 18. In someembodiments, the switches and/or actuators can be coupled to anelectromechanical system to rotate and/or translate the movement controlsystem 10130.

FIG. 8A illustrates a perspective view of an example gimbal system 12000for an imager 18, the gimbal system 12000 coupled to a viewing assembly9 comprising 2 pairs of oculars, 12005, 12010. The first pair of oculars12005 is configured to adjust its orientation about an axis 12007. Thesecond pair of oculars 12010 is configured to adjust its orientationrelative to an axis 12012 as well as relative to the first pair ofoculars 12005. For example, the second pair of oculars 12010 can beoriented such that users of the two pairs of oculars face one anotherwhen using the viewing assembly 9, such as when a doctor and anassistant are on opposite sides of a patient. The second pair of oculars12010 can also be oriented such that it is at about 90 degrees relativeto the first pair of oculars 12005. Other relative orientations are alsoavailable, such as any value between about 15 degrees to about 180degrees between the first and second pairs of oculars 12005, 12010.

The gimbal system 12000 can include a pair of handles 12002 a, 12002 bto allow a user to change the orientation of the imager 18. Asillustrated, the user can adjust the pitch and yaw of the imager 18using the handles 12002 a, 12002 b.

FIG. 8B illustrates a perspective view of a second example gimbal system12020 for an imager 18, the gimbal system 12020 coupled to a viewingassembly 9. The support structure 12024 for the imager 18 is configuredto allow a user to change the orientation and position of the imager 18.

In some embodiments, the handles 12022 a, 12022 b are used tomechanically alter the position and/or orientation of the imager 18. Incertain embodiments, the handles 12022 a, 12022 b are used as electroniccontrols to control one or more motor systems to orient and position theimager 18. In some embodiments, the handles 12022 a, 12022 b are aconvenience or comfort for a user, and other separate controls are usedto control the position and/or orientation of the imager 18 relative tothe viewing assembly 9.

In some embodiments, one or more handles 12022 can include controls forcontrolling features of the overall system. For example the handle caninclude one or more controls, e.g., button(s), for altering theillumination, zoom, focus, work distance, camera view provided,arrangement of camera views or any combination of these feature or otherfeatures in the alternative or in addition.

Although a gimbal system is shown, other types of systems for position(e.g., x, y, and/or z) and orienting (e.g., pitch, yaw, and possiblyroll), may be employed. In some embodiments, encoders or sensors providesignals with regard to the position and/or orientation of the camera. Insome embodiments, the gimbal or positioning and/or orientation systemmay include motors or actuators that can be controlled by controlelectronics. In some embodiments the control electronics can beconfigured to cause to gimbal or other positioning and/or orientationsystem to return to a preset position and orientation. Memory, may forexample be included that record certain preset positions and/ororientations. Such preset positions and/or orientations may be positionsand/or orientations for certain types of surgical procedures, forcertain surgeons, or combinations of both. Selection of the particularprocedure or indication of the particular surgeon may cause the gimbalor postion/orientation system to go to the appropriate preset positionstored in memory. Such preset positions and/or orientations may alsoinclude a storage position.

Other types of positioning and/or orientation systems may include ahexapod and/or an articulated arm.

In some embodiments, light weight material may be used to form thedisplay and/or console such as the housing. A honeycomb structure mayfor example be employed as a housing or cover.

The discussions above or elsewhere herein may be applicable to othertypes of cameras and positioning and/or orientation systems for othertypes of cameras including but not limited to one or more cameras on asurgical tool(s), one or more proximal cameras disposed outside apatient but within a close proximity to the patient and/or surgicalsite, such as 5 mm, 10 mm, 20 mm, 25 mm to 30 mm, 40 mm, 45 mm or anyranges therebetween. The distance of the proximal cameras to thepatient's body and or surgical site may be greater or less than thesevalues recited above.

Optical Systems for Displavs

FIGS. 9A-9B illustrate example display optical systems 11005 configuredto provide a view of displays 11010 through oculars (not shown) thatreceive light from the last lens 11015 in the display optical system11005. The display optical system 11005 forms an exit pupil at or nearthe entrance pupil of the surgeon binoculars. These pupils are closelymatched, for example, in size and shape. In some embodiments, the exitpupil of the display optical system 11005 can be the same size orsmaller than the entrance pupil of oculars used to view the display. Theoculars form an exit pupil that is matched (e.g., in size and shape) tothe entrance pupil of the surgeon's eye(s). In some embodiments, thedisplay optical system 11005 is configured to produce a beam that has arelatively constant cross-section between the first lens element 11012and the last lens element 11015, where the cross-section is relativelysmall. Advantageously, this allows the display optical system 11005 tobe included in a relatively small or compact package and use relativelysmall optical elements. In some embodiments, the last lens 11015collimates the beam leaving the display optical system 11005. Thetermination of the rays shown in FIG. 9A to the left of lens 11015 isthe exit pupil of the display optical system 11005. In some embodiments,the exit pupil of the display optical system 11005 is configured to bethe same size or smaller than, and positioned at the same location, asan entrance pupil of a binocular viewing assembly configured to allow auser to view the display 11010.

The lenses in the display optical system 11005 form a highlycolor-corrected view of the display by forming the exit pupil in aposition favorably disposed for the user and the binoculars. Acombination of singlets and bonded lenses provide such correction. Thedisplay optical system 11005 may be designed to provide such correctionwhile keeping a small beam column or ray bundle, which permits addingmirrors and obtaining a compact package. In various embodiments,producing an undistorted image can be difficult without such a group oflenses designed properly to provide such correction. This correctionincludes both color correction as well as distortion correction.

The display optical system 11005 advantageously allows a relativelysmall, compact lens assembly to provide a view of a relatively largedisplay 11010. The display optical system 11005 can be configured towork with displays 11010 of varying sizes, including, withoutlimitation, displays with a diagonal that is less than or equal to about0.86 in. (22 mm), at least about 0.86 in. (22 mm) and/or less than orequal to about 10 in., at least about 1 in. and/or less than or equal toabout 9 in., at least about 2 in. and/or less than or equal to about 8in., or at least about 4 in. and/or less than or equal to about 6 in.The display may, for example, have a diagonal of about 5 inches or about8 inches in some embodiments. The display can be configured to have arelatively high pixel count (e.g., 1920×1080 pixels, 1280×720 pixels,3840×2160 pixels, etc.). The total optical path length of the displayoptical system 11005 can be less than or equal to about 9 in., at leastabout 9 in. and/or less than or equal to about 20 in., at least about 10in. and/or less than or equal to about 19 in., at least about 14 in.and/or less than or equal to about 18 in. The display optical system11005 can include lenses, mirrors, prisms, and other optical elementsconfigured to direct and manipulate light along an optical path. Thedisplay optical system 11005 can be used in conjunction with a primarydisplay, a surgeon display, an assistant display, possibly otherdisplays, or any combination of these.

The example display optical system 11005 illustrated in FIG. 9A has atotal optical path length of about 16.2 in. (412 mm). It is configuredto provide an image of a 5 in. display 11010. The display optical system11005 can include a lens 11012 configured to direct the light from thedisplay 11010 along a path wherein light from the display 11010 isdirected along a path with a relatively narrow cross-section. In variousembodiments, the light received from the display is initiallysubstantially reduced in beam size for example by the lens 11012 orlenses closest to the display and a more narrow beam is produced. Incertain embodiments, for example, the lens 11012 or lenses closest tothe display collect light at an angle (half angle) in excess of 20°,25°, 30° and reduce the beam size of the light. This design isadvantageous because it allows for the elements in the display opticalsystem 11005 to be relatively small and compact. In some embodiments,the cross-section of the optical beam after the lens 11012 in thedisplay optical system 11005 can be configured to be relativelyconstant. This configuration allows folding or redirecting mirrorspresent in the optical path to remain small.

FIG. 9B illustrates a binocular display optical system 11005 configuredto provide a view of stereo displays 11010 a, 11010 b through a pair ofoculars. The binocular display optical system 11005 can be based on theoptical design illustrated in FIG. 9A, and can include one or moreelements 11014 in the optical path before the lens 11012 to reduce thephysical size of the optical system while maintaining the length of theoptical path. These elements can include mirrors, prisms, and/or otheroptical elements configured to redirect the light from the displays11010 a, 11010 b to the lens 11012. In some embodiments, the elements11014 include curved mirrors which redirect the optical path andconverge the rays from the displays 11010 a, 11010 b. In someembodiments, the elements 11014 include mirrors or prisms (for examplethat may have planar reflecting surface) that do not substantiallyaffect the convergence of the light rays, but redirect the optical path.In some embodiments, because of the shape of the beam incident on thereflective surface, for example, mirror, the reflective surface orcross-section of the mirror is non-circular, and is, for example,elliptical. Accordingly, in various embodiments the cross-section of themirror or other reflective surface is possibly being longer in onedirection than in another, for example, orthogonal direction. Theseelements may fold the optical path to provide for a more compact system.Such a system may therefore have an optical path length from display toocular that is longer than the length and/or width of the viewingplatform of the combination thereof.

In some embodiments, the display optical system 11005 can include atleast four mirrors, or less than or equal to four mirrors. In certainimplementations, two mirrors can be used to fold the optical path fromthe display 11010 to the exit pupil, the two mirrors positioned betweenthe first lens 11012 and the display 11010. In some embodiments, thedisplay optical system 11005 includes at least four lenses or less thanor equal to four lenses.

In some embodiments, the display optical system 11300 can include atleast four baffles or less than or equal to four baffles. In certainimplementations, four baffles can be included in the optical pathbetween the first lens and the display 11310. In some implementations,two mirrors can be included in the optical path between the first lensand the display 11310. In some embodiments, the optical path caninclude, in order from the display 11310, a first baffle, a firstmirror, a second baffle, a second mirror, and a third baffle prior tothe first lens.

In some embodiments, the display optical system can include binocularshaving an optical power of about 10×. The binoculars can have a field ofview of about 80 degrees to about 90 degrees. The binoculars can beconfigured to provide a field of view that is relatively wide (e.g.,panoramic) without producing a noticeable “kidney bean effect.” Thebinoculars can also be configured to provide a view of the displaywithout viewing the field stop. In some embodiments, the binoculars havea focal length of about 10 mm. The display optical system can include afield stop in the oculars. The display optical system can include acircular exit pupil with rectangular baffles.

The optics of the display optical system can include one or more opticalelements configured to output collimated rays. In some embodiments, theoptical elements, however, can be configured to not produce collimatedlight within the lens train of the display optical system. Byincorporating a converging lens near the display in the display opticalsystem, the light tube can be configured to be relatively small comparedto the size of the display. This can allow the size of the viewingassembly to be relatively compact.

In some embodiments, the display can be a curved surface, for exampleeither a projection display or recent generation of flexible LCD or OLEDdisplays having high-resolution (e.g., in excess of 300 ppi). A curveddisplay may provide two advantages: the imaging optics for the displaycan be less complex than for flat panels, and the cone or numericalaperture of each picture element in the display can be directed towardsthe viewing optics and in the periphery of the display, therebyproviding a brighter image less subject to vignetting.

In some embodiments, the display can be a volumetric display comprisingtwo or more transmissive display panels having a single backlightwherein the transmissive display panels are stacked to provide differentplanes of focus for a surgeon. The transmissive displays can be activematrix liquid crystal displays (“AMLCD”) or other types of transmissivedisplays. The backlight can be a fluorescent lamp, LEDs, or othersuitable light source. By having displays positioned in different focalplanes, image data from different focal planes may be presented to thesurgeon with relatively less image processing and/or compressioncompared to a system which combines data from multiple focal planes intoa single image. In some embodiments, a number of cameras can bepositioned at varying depths or having varying focal distances such thatthe displays at different focal planes are configured to display imagedata from cameras positioned or focused at different depths to create adisplay that assists the surgeon in identifying positions of featureswithin displayed images.

The display can show, as an overlay, pre-operative CT, MR, or other 3Dimage datasets from, for example, conventional surgical navigationsystems (e.g., the Medtronic StealthStation or Treon, Stryker SurgicalNavigation System, or Brainlab, among others). In various embodiments,in addition to images, the display can additionally provide numericaldata and/or text. For example, in various embodiments, the display canoverlay information such as distance or tool measurements, transparenttool renderings, camera identification information (e.g., the portion ofthe composite image attributable to a specific optical sensor maygenerate an identifying border around that portion), up/downorientation, elapsed time, and/or one or more still images captured fromone or more optical sensors from a previous time in the operation Thetracking system can provide 5-DOF (degrees of freedom) or 6-DOF positionand orientation information to conventional surgical navigation systems.Other information, graphic, alpha numeric, or otherwise, can beprovided.

The tool image can be magnified with respect to the wide-field viewimage, and change in image scaling will occur as the tool is moved inand out. In some embodiments, a visual metaphor for embodiments of thedisplay is that of a hand-held magnifying glass for inspecting and doingwork on a smaller region of a larger workpiece, while seeing the largerworkpiece with lower magnification (if any) in more peripheral regionsof the visual field to provide situational awareness. Tool images, forexample, can be superimposed on the background image thereby blockingthat portion of the background image. In various embodiments, the toolimages may be stereo.

FIG. 10 is a schematic illustration of a surgical visualization systemand an assistant display. In some embodiments, a separate assistantdisplay may be provided for use by a surgical assistant or observer. Asillustrated in FIG. 10, the assistant display 10035 comprises abinocular viewing platform 10036 for the assistant that includes oculars10039 mounted on a lockable articulated arm 10037, which extends from asupport post 10041. For example, the assistant 10031 and surgeon 10029may be positioned on opposite sides of the patient 10033, as in theillustrated arrangement. In such an arrangement, the image provided inthe assistant display 10035 may be rotated 180 degrees with respect tothat provided to the surgeon 10029. The assistant may be at otherlocations, for example, in other procedures. The assistant may, forexample, be located at a location 90 degrees with respect to thesurgeon, as opposed to 180 degrees with respect to the surgeon.Likewise, the image provided in the assistant display 10035 may berotated 90 degrees with respect to that provided to the surgeon 10029.Similarly the image may be reoriented as needed, possibly based on thelocation/position and perspective of the assistant. Additionally, theassistant display can be provided with any of the features describedelsewhere herein.

In some embodiments fluorescence images can be collected and displayed.These fluorescence images may be viewed superimposed on images of thesurgical site not based on fluorescence. Cameras that image in differentwavelengths, such as infrared, could image the surgical site or objectscontained therein. In some embodiments, features could be made tofluoresce, for example, by injecting fluorescent chemical andilluminating the area with light that will induce fluorescence. Forexample, in certain embodiments anatomical features may containfluorescent dye that fluoresces, for example, when exposed to shortwavelength radiation such as UV radiation. Such a technique may beuseful to identify and/or highlight the location and/or boundaries ofspecific features of interest such as tumors, etc. The fluorescence orother wavelength of interest may be detected by the one or more camerasimaging the surgical field such as one or more camera providing asurgical microscope view or one or more cameras on a surgical toolproviding a surgical tool view. For example, an optical detector that issensitive to the wavelength of the fluorescent emission may be employedto view the fluorescent image. In some embodiments, the wavelength offluorescent emission is in the infrared. In certain embodiments sensorssensitive to different wavelengths may be employed. In particular, oneor more sensors sensitive to the fluorescing wavelength (e.g., IR) maybe used in conjunction with one or more sensors not sensitive or lesssensitive to the fluorescing wavelength but sensitive or more sensitiveto other useful wavelengths (e.g. visible light). Light can be collectedand distributed to both types of detectors for example using abeamsplitter such as a wavelength dependent beamsplitter that reflectsone wavelength and passes another. The fluorescent and non-fluorescentimages can be recorded by the respective sensors. In some embodiments,the fluorescent and non-fluorescent images can be superimposed whendisplayed on electronic displays that receive image data from both typesof sensors. In various embodiments, the cameras, including fluorescentand/or non-fluorescent cameras, that provide a surgical microscope view,a surgical tool view (e.g., from a camera on a tool), or other view ofthe surgical site, may comprises stereo cameras and the displays maycomprise stereo displays.

In some embodiments, images produced by fluorescence or otherwavelengths of interest are superimposed on one or more images fromother camera(s). Filtering could be provided to remove unwantedwavelengths and possibly increase contrast. For example, the filter canbe used to remove excitation illumination. In some embodiments, emissionimage content, (e.g., fluorescing tissue) can be parsed and superimposedon image content that is not emitting (e.g., tissue that is notfluorescing), or vice versa.

In some embodiments, IR fluorescence images are superimposed over non-IR(e.g. visible) images. Other wavelengths such as other fluorescencewavelengths may be employed. In various embodiments, such as where thefluorescing wavelength is not visible (e.g., for fluorescence in theinfrared), an artificial color rendition of the fluorescing content canbe used in place of the actual fluorescing color so as to enable thefluorescing tissue to be visible.

FIG. 11 schematically illustrates an example medical apparatus inaccordance with certain embodiments described herein. The medicalapparatus 2100 can comprise a display (or display portion) 2110, aplurality of cameras 2120, and one or more processors 2130. Theplurality of cameras 2120 can include at least one first camera 2121 aconfigured to image fluorescence in a surgical field, and at least onesecond camera 2122 a configured to produce a non-fluorescence image ofthe surgical field. The processor 2130 can be configured to receiveimages from the plurality of cameras 2121 a, 2122 a, and to display onthe display 2110 a fluorescence image from the at least one first camera2121 a and to display on the display 2110 the non-fluorescence imagefrom the at least one second camera 2122 a. As shown in FIG. 11, theprocessor 2130 can advantageously include a plurality of processors 2131a, 2132 a, e.g., a separate processor for each camera within theplurality of cameras 2120. For example, at least one first processor2131 a can be configured to receive an image from at least one firstcamera 2121 a and to display on the display 2110 a fluorescence image.In addition, at least one second processor 2132 a can be configured toreceive an image from at least one second camera 2122 a and to displayon the display 2110 the non-fluorescence image.

The display 2110 can be a primary display, a surgeon display, anassistant display, possibly other displays, or any combination of these.The display 2110 can include a display portion, a display, or displaydevice as described herein. For example, in some embodiments, thedisplay 2110 can include a display (or display portion) to be viewedthrough one or more oculars, e.g., a display within the viewing platform9 of the surgical viewing system 1 shown in FIGS. 1, 2, 3A, 4A and 4B.The display (or display portion) could be within a housing. In otherembodiments, the display 2110 can include a display mounted on a displayarm from the ceiling or on a post, e.g., a display device 13 on displayarm 5 of the surgical viewing system 1 shown in FIG. 1 or be mounted onthe wall. In various embodiments, such displays comprise panel displayshaving a length of, for example, be between 15-70 inches, or larger orsmaller.

In various embodiments, the plurality of cameras 2120 can include acamera to provide a surgical microscope view of the surgical field. Insome embodiments, the plurality of cameras 2120 can include a cameradisposed on a surgical tool or on another medical device. The pluralityof cameras 2120 can include at least one first camera 2121 a and atleast one second camera 2122 a configured to form a left-eye view of thesurgical field. The plurality of cameras 2120 can also include at leastone first camera 2121 b and at least one second camera 2122 b configuredto form a right-eye view of the surgical field. In some embodiments, theleft and right-eye views are for stereoscopic viewing of the surgicalfield and the cameras can be angled to provide desired convergencemimicking the human eye. One or more cameras 2121 a, 2121 b, 2122 a,and/or 2122 b of the plurality of cameras 2120 can include opticalassemblies as described herein. For example, one or more cameras 2121 a,2121 b, 2122 a, and/or 2122 b can include a turning prism 54, a lenstrain 55, and/or a sensor 56 as shown in FIG. 5A.

As described herein, for the left-eye view, the at least one firstcamera 2121 a can be configured to image fluorescence in a surgicalfield, and the at least one second camera 2122 a can be configured toproduce a non-fluorescence image of the surgical field. Similarly, forthe right-eye view, the at least one first camera 2121 b can beconfigured to image fluorescence in a surgical field, and the at leastone second camera 2122 b can be configured to produce a non-fluorescenceimage of the surgical field.

In some embodiments, the first camera 2121 a and/or 2121 b can besensitive to infrared wavelengths, ultraviolet wavelengths, or otherfluorescence wavelengths. For example, an optical detector, e.g., sensor56 or an array of sensors, of the first camera 2121 a and/or 2121 b canbe sensitive to fluorescence wavelengths. In some embodiments, the firstcamera 2121 a and/or 2121 b sensitive to fluorescence wavelengths caninclude an infrared, ultraviolet, or other fluorescence light source. Insome embodiments, illumination using an optical fiber can be used toprovide pump radiation to induce fluorescence. In some embodiments, afilter may be used to selectively direct fluorescence wavelengths to thefirst camera 2121 a and/or 2121 b sensitive to fluorescence wavelengths.In some embodiments, the second camera 2122 a and/or 2122 b may not besensitive to fluorescence wavelengths.

In some embodiments, the processor 2130 can be configured to superimposethe fluorescence image over the non-fluorescence image. In otherembodiments, the processor 2130 can be configured to superimpose thenon-fluorescence image over the fluorescence image. In variousembodiments, the processor 2130 can electronically process andsynchronize the fluorescence and non-fluorescence images together. Forexample, the processor 2130 can read, align, and combine together theimages.

The processor 2130 can include a general all-purpose computer and insome embodiments, a single processor may drive both the left and rightdisplay portions 2110. However, various embodiments of the medicalapparatus 2100 can include separate processing electronics for theleft-eye and right-eye views. Such separate processing for the left andright channels can be advantageous over a processor with singleprocessing electronics or the general all-purpose computer since time iscritical in surgical procedures. For example, in some embodiments,having separate dedicated processing electronics for each channel canprovide pure parallel processing, which results in faster processing ofimages, thereby reducing latency. In addition, addressing a failure of ageneral all-purpose computer may entail rebooting of the computer andinvolve some downtime. Furthermore, with separate processing electronicsin left-eye and right-eye view channels, if one of the processingelectronics were to fail, the processing electronics in the otherchannel can continue to provide images to the surgeon. Such redundancycan also be incorporated into a monocular viewing system. For example,in some embodiments of a monocular viewing system, two channels similarto a binocular viewing system can be provided. Images for the monocularviewing system can be split into each channel, with each channel havingits own processing electronics.

Furthermore, in some even more advantageous embodiments, as shown inFIG. 11, the medical apparatus 2100 can include separate processing foreach camera within each channel to further increase processing of imagesand reduce latency. For example, for the left-eye view, processor 2131 acan be configured to receive an image from camera 2121 a and to displayon the display 2110 a fluorescence image from camera 2121 a. Processor2132 a can be configured to receive an image from camera 2122 a and todisplay on the display 2110 the non-fluorescence image from camera 2122a. The fluorescence and non-fluorescence images can be superimposedoptically on the display 2110. Similarly, for the right-eye view,processor 2131 b can be configured to receive images from camera 2121 band to display on the display 2110 a fluorescence image from camera 2121b. Processor 2132 b can be configured to receive images from camera 2122b and to display on the display 2110 the non-fluorescence image fromcamera 2122 b. The fluorescence and non-fluorescence images can besuperimposed optically on the display 2110.

In certain embodiments, each of the separate processing electronics canbe configured for image manipulation, e.g., to receive image data,process the image data, and output the images for display. For example,each of the processing electronics can be configured to receive one ormore user inputs, receive one or more input signals corresponding toimages from one or more cameras, and/or select which image to display.Each of the processing electronics can also resize, rotate, orreposition the selected image based at least in part on one or more userinputs or provide any combination of these operations. The processingelectronics can also produce one or more output signals to drive one ormore displays to produce one or more images. For example, eachprocessing electronics can include a microprocessor, a fieldprogrammable gate array (FPGA), or an application specific integratedcircuit (ASIC). Each processing electronics can also include a graphicsprocessing unit (GPU) and random access memory (RAM). The processingelectronics can also control the color balance, brightness, contrast,etc. of the one or more images or provide any combination of theseoperations.

In some embodiments, instead of superimposing fluorescence andnon-fluorescence images, an image at a first wavelength range can besuperimposed with an image at a second wavelength range. For example,one or more sensors can capture a first image at a first wavelengthrange, and one or more sensors can capture a second image at a secondwavelength range. The first and second images can be superimposedoptically as disclosed herein. As another example, the image at a firstwavelength can be provided by narrow band imaging instead offluorescence imaging. For example, a filter in some embodiments canallow imaging with the use of ambient light at blue (about 440 to about460 nm) and/or green (about 540 to about 560 nm) wavelengths for theimage at the first wavelength. Imaging at or near these wavelengths canimprove visibility of features since the peak light absorption ofhemoglobin occurs at these wavelengths. The image at the secondwavelength can be provided without narrow band imaging (e.g., use ofambient light without a filter).

In further embodiments, the plurality of cameras 2120 can includedifferent cameras for multiple views of the surgical site instead of orin addition to cameras mainly for imaging at different wavelengths. Forexample, in some embodiments, the plurality of cameras 2120 can includea camera providing a surgical microscope view, a camera disposed on asurgical tool (e.g. cutting tool), and a camera disposed on anothermedical device to provide different views of the surgical site or anycombination thereof. Some embodiments can also include a switch orswitching module to determine which views are to be displayed, forexample, as superimposed, overlapping, adjacent, stereo or as amonocular view, etc. One or more image could also be from other sources,e.g., a data file, a computed tomography (CT) scan, a computer aidedtomography (CAT) scan, magnetic resonance imaging (MRI), an x-ray,ultrasound imaging instrument, etc.

FIG. 12A schematically illustrates another example medical apparatus inaccordance with certain embodiments described herein. Some suchembodiments can also advantageously decrease the time to produce animage for viewing, which can be important in certain surgicalprocedures. For example, the medical apparatus 2200 can include aplurality of displays (or display portions), a plurality of cameras, andone or more beam combiners. As shown in FIG. 12A, to form a left-eyeview, the plurality of cameras can include at least one first camera2221 a configured to produce a fluorescence image onto a first display2211 a and at least one second camera 2222 a configured to produce anon-fluorescence image onto a second display 2212 a. In someembodiments, the cameras 2221 a, 2222 a can produce the images onto theplurality of displays 2211 a, 2212 a, e.g., with a processor. However,in such embodiments, an electronic processor need not perform thecombining of images. A beam combiner 2230 a can be configured to receivethe fluorescence and non-fluorescence images from the first 2211 a andsecond 2212 a displays and to combine or superimpose optically thefluorescence and non-fluorescence images for left-eye viewing, e.g.,within a housing through an ocular or on a display device.

As shown in FIG. 12B, to form a right-eye view, the plurality of camerascan also include another first camera 2221 b configured to produce afluorescence image onto another first display 2211 b and another secondcamera 2222 b configured to produce a non-fluorescence image ontoanother second display 2212 b. In some embodiments, the cameras 2221 b,2222 b can obtain images that can be viewed on the plurality of displays2211 b, 2212 b, for example, using processing electronics. However, insuch embodiments, an electronic processor need not perform the combiningof images. Combining images into a single display, for example, mayinvolve a central processor working to a single clock. Instead, a beamcombiner 2230 b can be configured to receive the fluorescence andnon-fluorescence images from the first 2211 b and second 2212 b displaysand to superimpose the fluorescence and non-fluorescence images forright-eye viewing, e.g., within a housing through an ocular or on adisplay device. By optically combining multiple images from varioussources, with or without stereo convergent characteristics, differentlytimed imaging signals (for example, with different frame rates) ordifferently resolved imaging signals (wherein, for example, the pixelcount can be the full pixel count or a subset of the full pixel count)can be sent to their respective displays and viewed by the viewer asoptically compatible without the need for them to be electrically madetime or resolution compatible. For example, screens of differentresolution such as a 5 inch screen having 1080×1920 pixels may becombined with a display for a fluorescence image wherein the display has800×520 pixels. The different spatial resolution (and/or size) of thetwo displays are combined optically by the beam combiner and processedby the eye. Such an example could apply for either or both the right eyeand/or left eye. Other variations in the displays are possible such asdisplays with different timing. For example, a high definition (HD)display operating at 60 frames per second can be optically combined withimages from a fluorescence camera operating at 30 frames a second.Instead of speeding up or slowing down one of the signals with respectto the other in a single processor, images from the separate displayshaving different timing can be combine optically with a beamsplitter orbeamcombiner. More than two displays having such different features(e.g., size, resolution, timing) can be combined optically in thismanner for either or both the left eye and/or right eye.

In various embodiments, the beam combiner 2230 can include abeamsplitter (e.g., a 45 degree or other angle splitter used inreverse), a dichroic beamsplitter, a prism, or other optical structureto combine the beams. As an example, a beam combiner 2230 a can beplaced within the left-eye optical path to receive the fluorescence andnon-fluorescence images from the first 2211 a and second 2212 a displaysand to superimpose the fluorescence and non-fluorescence images forleft-eye viewing, e.g., within a housing through an ocular or on adisplay device. Similarly, another beam combiner 2230 b can be placed inthe right-eye optical path to receive the fluorescence andnon-fluorescence images from the first 2211 b and second 2212 b displaysand to superimpose the fluorescence and non-fluorescence images forright-eye viewing. Some embodiments can further include imaging optics(e.g., an optics assembly) disposed to collect light from the displaysto enable the images to overlap. The imaging optics can be configured toform images at infinity. The imaging optics can be configured to be seenby an observer as though the viewer is seeing the displays at infinitywith relaxed accommodation. FIG. 12C schematically illustrates a topview of an embodiment of a medical apparatus incorporating the exampleleft and right assemblies from FIGS. 12A and 12B.

In some embodiments, instead of superimposing fluorescence andnon-fluorescence images, an image at a first wavelength range can besuperimposed with an image at a second wavelength range. For example, afirst camera 2221 a can produce a first image at a first wavelengthrange onto a first display 2211 a, and a second camera 2222 a canproduce a second image at a second wavelength range onto a seconddisplay 2212 a. The beam combiner 2230 a can optically superimpose thefirst and second images. As another example, the image at a firstwavelength can be provided by narrow band imaging instead offluorescence imaging, and the image at the second wavelength can beprovided without narrow band imaging as described herein.

In addition, images from two different cameras of the same orsubstantially the same wavelength, but having other properties can besuperimposed. For example, one image could be a natural image of tissue,and another view could be an unnatural image (e.g., an image with falsecolor or an image with exaggerated or extreme contrast). In someembodiments, such superimposed images can advantageously show marginsbetween healthy and unhealthy tissue. The example embodiments of themedical apparatuses shown in FIGS. 12 and 12A-12C can also be modifiedto produce a composite image of two or more images. FIG. 14A illustratesa schematic of an example composite image 2500, where a first (e.g., abackground) image 2501 is produced on a first portion 2511 of thecomposite image 2500, and a second (e.g., a picture-in-picture (PIP))image 2502 is produced on a second portion 2512 of the composite image2500. In some embodiments, the images can include a fluorescence imageand a non-fluorescence image. However, in other embodiments, the imagesare not necessarily fluorescence and non-fluorescence images. Forexample, one image can be a surgical microscope view of the surgicalfield from a camera producing the surgical microscope view. The otherimage can be the image of the surgical field from a camera disposed on asurgical tool (e.g. cutting tool) or other medical device. One or moreimage could also be from sources other than cameras, e.g., a data file,a computed tomography (CT) scan, a computer aided tomography (CAT) scan,magnetic resonance imaging (MRI), an x-ray, ultrasound imaginginstrument, etc. FIG. 14B schematically illustrates a front view of anembodiment of a medical apparatus incorporating the example left andright assemblies from FIG. 11 or 12A-12C to produce a composite image oftwo or more images for both left and right eyes.

Referring to the example embodiment shown in FIG. 11, for the left-eyeview, the first camera 2121 a can be a camera producing a surgicalmicroscope view, and the second camera 2122 a can be a camera disposedon a surgical tool (e.g. cutting tool) or other medical device.Similarly, for the right-eye view, the first camera 2121 b can beanother camera producing a surgical microscope view, and the secondcamera 2122 b can be another camera disposed on a surgical tool (e.g.cutting tool) or other medical device. For each eye's view, the firstcamera 2121 a, 2121 b can produce the background image 2501 of thecomposite image 2500, and the second camera 2122 a, 2122 b can producethe PIP image 2502 in the composite image 2500. For the left-eye view,the processor 2131 a can be configured to receive an image from thefirst camera 2121 a and to display on the display 2110 the image as thebackground image 2501 of the composite image 2500. In addition, theprocessor 2132 a can be configured to receive an image from the secondcamera 2122 a and to display on the display 2110 the image as the PIPimage 2502 of the composite image 2500. For the right-eye view, theprocessor 2131 b can be configured to receive an image from the firstcamera 2121 b and to display on the display 2110 the image as thebackground image 2501 of the composite image 2500. In addition, theprocessor 2132 b can be configured to receive an image from the secondcamera 2122 b and to display on the display 2110 the image as the PIPimage 2502 of the composite image 2500. As shown in FIG. 13B, theposition of the PIP image 2502 in the composite image 2500 can be in thesame or different location from that illustrated in the figures.Additional cameras or sources can also be used to produce a multiple PIPimages.

Referring to the example embodiment shown in FIGS. 12A-12C, a beamcombiner 2230 a, 2230 b can be placed within each eye's optical path toproduce the composite image 2500. In some embodiments, the backgroundimage from a camera can be resized or the row count of pixels of thebackground image can be reduced. For example, the background image canbe resized from the full frame to the size of the first portion 2511(e.g., about ½, ⅔, ¾, etc., or any range therebetween) of the compositeimage 2500. The beam combiner 2230 a, 2230 b in each eye's optical pathbetween the viewer and the displays can superimpose the background imagewith a PIP image such that the background image appears on the firstportion 2511 of the composite image 2500, and the PIP image forms withinthe remaining portion 2512 (e.g., about ½, ⅓, ¼, etc., or any rangetherebetween) of the composite image 2500. In some embodiments, theremaining portion 2512 can include a border 2513 having a thickness(e.g., 1%, 2%, 3%, 5%, 100%, or 15%, of the width of the image, or anyrange therebetween) surrounding the PIP image 2502 to help prevent theviewer from seeing similar types of images as being falsely contiguous(e.g., similar types of tissues from multiple sources).

With reference to FIG. 12A, an example illustration using the left-eyeview will be provided. The example illustration can also apply to theright-eye view in certain embodiments. For example, a first camera 2221a for providing a surgical microscope view can provide the backgroundimage on a first display 2211 a, and a second camera 2222 a disposed ona surgical tool or other medical device can provide the smaller image ona second display 2212 a. The beam combiner 2230 a can produce thebackground image from the first display 2211 a as the first portion 2511(e.g., about ⅔) of the composite image 2500. The beam combiner 2230 acan also combine the PIP image from the second display 2212 a as part ofa second portion 2512 (e.g., about ⅓) of the composite image 2500. Asshown in FIG. 13A, the background image 2501 can be produced in themajority (e.g., about ⅔) of the composite image 2500. The PIP image 2502can be produced as part of, e.g., within the remaining portion 2512(e.g., about ⅓) of the composite image 2500.

The display 2211 a for the background image can be a 5″ display. Thesmaller PIP image from the second camera 2222 a can be displayed on asmaller panel viewed off from the beam combiner 2230 a, or could bedisplayed on a 5″ display using only a portion of the display (e.g.,about ⅓ of the display or about part of ⅓ of the display). Afterproperly baffling the optical pathways, the viewer can see the smallerimage 2502 adjacent the background image 2501 as though it were apicture-in-picture.

The beam combiner 2230 can also produce additional PIP images from otherdisplays as part of the composite image 2500. For example, multipleimages (e.g., two, three, four, five, six, nine, twelve, etc., or anyrange therebetween) from multiple displays (e.g., two, three, four,five, six, nine, twelve, etc., or any range therebetween) can be viewedfor each eye's view by using one or more beam combiners 2230.

In some embodiments, the smaller images can be superimposed with a dark(e.g., black) or light (e.g., clear) border to prevent the viewer fromseeing similar images as being falsely contiguous (e.g., similar typesof tissues from multiple sources). For example, after resizing thebackground image (e.g., to about ⅔ size), the remaining portion (e.g.,about ⅓) of the image can be left black. The smaller images from otherdisplays can be superimposed onto the black portion of the backgroundimage such that the images do not appear falsely contiguous. Inaddition, the border can help facilitate the beam combiner 2230arrangement, making the alignment less critical in some embodiments. Invarious embodiments, the border to can have a thickness of between of 2%to 5% or 3% to 10% of the width of the images or larger. In someembodiments, the smaller images could be superimposed onto thebackground image. For example, the background image could includeadditional superimposed or overlapping images. Some embodiments caninclude a switch or switching module to determine which image to bedisplayed. For example, the background image could be switched off andnot be displayed so that a different image(s) can be displayed in thefirst portion 2511 of the view 2500.

As described herein, two images can form a composite image. For example,two non-latent images, e.g., two real-time images of the surgical field,can form a composite image. In various embodiments, when the horizontalline of sight is maintained, merging of images is possible. In addition,one non-latent image (e.g., a real-time image of the surgical field) andone latent image (e.g., a data file, a CT scan, a CAT scan, an MRI, anx-ray, ultrasound image, etc.) can form a composite image. In variousembodiments, the latent and/or non-latent images can be seenindividually by both eyes; and one or more of the images can haveconvergence information for a stereo or 3D effect. For example, theimages can be displayed on 2D displays that represent 2D images from theinput cameras. However, with convergence information, the brain canallow the eyes to see a 3D image.

FIG. 13B1 shows an illustration of an example medical apparatusaccording to certain embodiments described herein. FIG. 13B1-a shows alarger view of the side view of FIG. 13B1; and FIG. 13B1-b shows alarger view of the front view of FIG. 13B1. Example dimensional (e.g.,in millimeters) values are provided. However, the dimensions are notparticularly limited. As shown in FIG. 13B1-a, the example embodiment ofthe medical apparatus 3000 includes an imaging optics assembly 3003, abeam combiner 3004, a first display 3005 a, and a second display 3006 a.In some embodiments, the first display 3005 a can be the same size asthe second display 3006 a. In some embodiments, one of the displays 3005a can be a combination of smaller displays with a corresponding outerdimension equal to the other display 3006 a. In some other embodiments,the first display 3005 a can have a different size than the seconddisplay 3006 a. In some embodiments, the imaging optics assembly 3003can include a mirror 3007 to relocate the optical path from the viewingoculars, chambers, ports or portals to a direction upwards (e.g.vertically) to the displays 3005 a, 3006 a. This can allow for theoptical path from the viewing oculars, chambers, ports or portals to beof a suitable length without moving the oculars, chambers, ports orportals further (e.g., horizontally) from the surgical site. In someembodiments, the imaging optics assembly 3003 can have a periscopedesign with a first mirror or reflector configured to direct the opticalpath from the ocular or portal upward or vertically and anotherreflector, possibly a partially reflecting partially transmittingbeamsplitter to reflect at least a portion of the optical pathhorizontally similar to a periscope. In some embodiments, the surgeoncan advantageously be positioned to view the surgical site and tomanipulate tools in an ergonomic way with the surgeon's armssufficiently close to the surgical site so as not to need to stretch hisor her arms, which can be especially uncomfortable for long surgicalprocedures. Accordingly, in some embodiments, 70% to 95% (e.g., 75% to90%) of the height of the viewing assembly 3000 can be located above theviewing oculars, chambers, or ports. In some embodiments, the horizontaldistance from the entrance of the ocular or portal to the display is notlarger than the vertical distance from the entrance of the ocular orportal to the display 3005 a.

As illustrated in the example embodiments, the distance between theocular or eye portal optics and the display where the distance is thevertical distance or the distance along an axis perpendicular to theoptical axis of the ocular or eye portal can be about 173 mm or betweenabout 100 mm and about 250 mm, between about 120 mm and about 225 mm, orbetween about 140 mm and about 200 mm or any range between any of thesevalues. The distance between the ocular or eye portal optics and thedisplay where the distance is the horizontal distance or the distancealong an axis parallel to the optical axis of the ocular or eye portalcan be less than the vertical distance. In some embodiments, thehorizontal distance is about 100 mm or between about 50 mm and about 200mm, between about 75 mm and about 150 mm, or between about 90 mm andabout 125 mm or any range between any of these values. In someembodiments, the horizontal distance is about 50% of the verticaldistance, or between about 40% and about 90% of the vertical distance,between about 50% and about 80% of the vertical distance, or betweenabout 60% and about 70% of the vertical distance or any range betweenany of these values. In some embodiments, the display housing of themedical apparatus has greater than or equal to about 50% of the displayhousing volume above the oculars or eye portals. In some embodiments,the display housing of the medical apparatus has greater than or equalto about 60% of the display housing volume above the oculars or eyeportals, greater than or equal to about 70% of the display housingvolume above the oculars or eye portals, greater than or equal to about80% of the display housing volume above the oculars or eye portals, orgreater than or equal to about 90% of the display housing volume abovethe oculars or eye portals, or greater than or equal to about 70% and/orless than or equal to about 95% of the display housing volume above theoculars or eye portals. As described herein, the displays can bepositioned above the center of the oculars or eye portals where abovethe center of the oculars or eye portals can be any position above aplane defined by the optical axis at the exit window (e.g., lens ortransparent window) from the display to the eye of the user in theocular or eye portal. When the plane defined by the optical axis at theexit window is horizontal (e.g., perpendicular to gravity), then anobject is above that plane when it is displaced along an axisperpendicular to the defined plane in a direction opposite the directionof gravity. If the display unit rotates to change the orientation of thedefined plane, then, in some instances, the relative orientation of thecomponents remains substantially fixed.

FIG. 13B1-c shows the example medical apparatus of FIG. 13B1 with an eye3001 viewing into an ocular 3002. The medical apparatus includes animaging optics assembly 3003, a beam combiner 3004, and first and seconddisplays 3005 a, 3006 a. As described herein, the first display 3005 acan display an image with a portion (e.g., bottom ¼ of the display) leftblack. The second display 3006 a can display another image with aportion (e.g., top ¾ of the display) left black. FIG. 13B1-c also showsan alternative example of a first display 3015 a with a ¼ corner leftblack, and a second display 3016 a with the remaining ¾ portion leftblack. Other examples are possible.

As described herein, each eye can merge the two images together suchthat the two displays appear as one. For example, the eye can form acomposite image including a background image and a PIP image. In variousembodiments, the predominant image (e.g., from the first display 3005 a)can be a non-latent image (e.g., a surgical microscope view of thesurgical field), while the supplementary view, supplementaryinformation, supplementary text, and/or supplementary overlays (e.g.,from the second display 3006 a) can be another non-latent image (e.g.,an image of the surgical field from a surgical tool) or a latent image(e.g., a data file, a CT scan, a CAT scan, an MRI, an x-ray, anultrasound image, etc.). In some embodiments, the two images can beseparated from each other by a black bar or the two images cansubstantially overlap (e.g., in the form of a fluorescence image or anear IR image that goes through a processing box).

FIG. 13B1-d shows an illustration of a beam combiner arrangement 3004, afirst camera 3008 a, and a second camera 3009 a. The first camera 3008 ais in communication with a first camera controller 3010 a, and thesecond camera 3009 a is in communication with a second camera controller3011 a. The first camera controller 3010 a can receive signals from thefirst camera 3008 a, and the second camera controller 3011 a can receivesignals from the second camera 3009 a. A graphical user interface (GUI)controller 3012 and switcher 3013 can communicate how to produce theimages on the displays 3005 a, 3006 a (e.g., which portions to leaveblack) with the first and second camera controllers 3010 a, 3011 a. Incertain embodiments of the medical apparatus, such processing does notintroduce latency for real-time images because such resizing is not alatency-causing activity or function. For example, in some embodiments,the processing can be performed by a video switcher and/or controllerand not by a single central processing unit. In various embodiments, thefirst and/or second camera controllers 3010 a, 3011 a can include imageprocessors and/or camera control units. In some embodiments, the firstand/or second camera controllers 3010 a, 3011 a can have modestfunctions. For example, in some embodiments, the first and/or secondcamera controllers 3010 a, 3011 a do not have to include computers. Asan example, some embodiments include two non-latent lines, signals, orchannels.

Certain embodiments of the medical apparatus can receive and displayimages based on the request from the GUI. For example, upon request,some embodiments can receive a latent image to be displayed. As anexample, a static image of an x-ray, a CAT scan, an MRI, or an imagefrom a computer can be received from a source, such as in a format likeDICOM (Digital Imaging and Communications in Medicine). A non-latentimage can also be received from another source. In some embodiments, thepredominant image can be the non-latent image. However, in someinstances, the predominant image can be the latent image. For example, asurgeon may wish to view the MRI. The surgeon would know that the MRI isnot in real-time. However, the surgeon may wish to have the real-timeimage of the surgical site in view. In some such instances, the surgeoncan switch, via the GUI, the predominant image to the MRI and keep theimage of the surgical site as a PIP image. Various embodiments of themedical apparatus do not include a direct optical connection between thetwo displays (e.g., a decoupled display system) and are not configuredin the same way as an operating room microscope.

Certain embodiments of the medical apparatus can include four displays(e.g., two displays for each eye). As disclosed herein, some embodimentscan include three or more displays for at least one eye. For example,instead of the second display being as equal to the first, the seconddisplay could be a matrix of four smaller screens from four sourcechannels. From a processor standpoint, a section of the display can bedesigned as non-latent, and can receive a non-latent image from anotherportion of the display or from another source. In some embodiments, twonon-latent real-time images can be imposed as a PIP view in one display,and the transformation can be performed by either the camera or thedisplay system, by multiple cameras or the display system, or by acontroller. In some embodiments, when two non-latent images aresuperimposed and/or overlaid, there can be varying degrees oftransparency. In addition, in some embodiments, the images can beregistered with both the left and right eyes. For example, the PIPimages for the left and right eyes can be placed in the same relativespace (e.g., upper right hand corner for both left and right eyes).

In some embodiments, a latent image can be displayed on one display andthe non-latent image on another display. In some embodiments, a latentimage can be inserted into a non-latent image. For example, the imagescan be processed through a computer or a computer-like device (e.g., asame computer with the same display). Certain embodiments of the medicalapparatus can include two or more displays to provide images (latentand/or non-latent) overlaid or integrated and merged in each eye. Theadvantage of certain embodiments is the ability to view images (latentand/or non-latent) from multiple sources, e.g., distal cameras, proximalcameras, external sources, etc.

In addition, sources with different formats can be combined. Forexample, one display can include an image from a digital source and theother display can include an image from an analog source. As oneexample, one camera can be a digital 3G SDI camera and the other cameracan be an analog camera. If one of the displays was set up for an analogformat and the other display was set up for a digital format, certainembodiments of a medical apparatus can mix formats without having toconvert them into a common format. Accordingly, certain embodiments of amedical apparatus can provide a way of merging digital and analogformats in the same view without translation. In various embodiments,dissimilar and/or incompatible imaging sources can be combined by theuser in each eye. In some embodiments, each eye's combined view mightdiffer only by virtue of the convergence angle at the source. As anexample, a near IR image can allow one to see within the tissue, e.g.,within 1 cm in some cases from the surface. The convergence angles canbe parallel, but lie outside the convergence angle of the visible image.The images on the display can be overlaid and displaced to match theconvergence depth.

FIG. 13C illustrates that the images are not restricted to PIP images.FIG. 13C shows, for example, a schematic of an example view 2600 ofmultiple images (e.g., from multiple sources) of the surgical fielddisposed adjacent to one another. For example, a first (e.g., abackground) image 2601 is produced on a first portion 2611 of the view2600, and a second (e.g., a smaller or of similar size) image 2602 isproduced on a second portion 2612 of the view 2600 such that the imagesdo not necessarily overlap one another or do not need to substantiallyoverlap, or one image does not need to be substantially contained withinthe other images. In such embodiments, the images can appear adjacent toone another or tiles in a manner that is not restricted to a PIParrangement. As described above, more than two images may be included,for example, tiled with respect to each other. Additionally, more thanone beam combiner and more than two displays may be employed in variousembodiments to combine images, for example, for the left eye (or for theright eye).

As described herein, some embodiments as shown in FIG. 12A-12C can, bythe use of beam combiners 2230, advantageously can reduce latency bydecreasing the time to produce an image for viewing. For example,multiple images can be tiled to view the multiple images from a varietyof sources as opposed to being aligned and combined using an imageprocessing technique that consumes computing power. In addition, anadvantage of additional displays in each eye's path in certainembodiments can present to the viewer superimposed images without thecomplexity of electrical registration and timing issues. In some suchembodiments, the brain can also merge the images if the additionaldisplays are reasonably aligned optically.

In some embodiments, the above combination of images can be secondarilyor additionally run through a central processor to produce a singlecomposite (e.g., tiled, or picture-in-picture, etc.) view for a singledisplay. Such a combination of differing sources may produce latency,however, the resultant image can be provided to other viewers in theroom besides the operating surgeon. Such viewers or observers, such asstaff, in the room may have less reason to require a low latencycombination of images however such viewers may still benefit from seeingthe surgery. For example, staff may be able to anticipate tools that thesurgeon will request upon viewing the progress of the surgery. A centralprocessor could produce stereo images for monitors plus goggles and/orauto stereoscopic monitors to be used by the ancillary medical personnel(e.g., staff).

In some embodiments, one or more of the separate signals directed to theseparate displays can be accessed for use such as for recording. In somesuch configurations, one of the outputs from the cameras that aresending video to the respective display(s) could be used separately asthe images are electronically separate. For example, the video signaldirected to one of the displays can be recorded separately without beingcombined with images from another display in the recording. A recordingdevice, such as a USB recorder or other storage device could beconnected to a port associated with that camera or display allowingrecording of the one or more images provided to that display. The usercould control the segment of image signal recorded or otherwise used.For example, a still image or a segment of video of any duration (e.g.,1, 2, 3, 4, 5, 10, 20 or more seconds or minutes, or any range betweenany of these values) could be recorded. Input from the surgeon may beprovided via a user interface to record for a certain duration such asfor 1 to 10, 20, or 30 seconds or 30 seconds to a minute or from 1 to 2,3, 5, or 10 minutes or longer. The signal that is recorded or otherwiseobtained and used could be obtained from the separate camera or displayfor any duration and accessed at different times. Such a feature may beprovided for one or more of the cameras and/or displays such that thesurgeon may select which camera (based on viewing the respective displayreceiving video from that camera) from which to draw the signal forrecording or other use without needing to obtain the signal (video)electronically combined with video displayed on the other displays. Eachcamera could, for example, have associated therewith a USB recorder or aport for such a recorder. As described above, the recording can be donelocally on any or all input signals. The user could therefore edit (andpossibly reorient and reposition) and/or emphasize a particular eventfor teaching or documentation purposes.

In various embodiments, the different displays have stages or platformsor are otherwise configured to be repositioned and/or reorientedmechanically to display for the surgeon respective video at differentpositions and orientations. In some embodiments, one or more of thedifferent displays could be moved physically with respect to one anotherby the user. The displays could be mounted so as to be movable and/orreoriented by the surgeon either by hand or electronically. The usercould thus move one display from one location on the left or top toanother location on the right or bottom (or vice versa) of the view seenby the user. The user can thus reposition so overlay is re-registered.This may be done mechanically and optical (with a beam combiner) ratherthan electronically (although the displays may be moved electronically).In some cases a camera may be zoomed in (or out) and the zoomed viewshown on the display may be repositioned (and/or reoriented) byphysically moving the display on which the zoomed image is shown withrespect to an image or information provided by another display. Thiscapability may allow the surgeon to selectively emphasize orde-emphasize certain views or inputs from cameras or other sourcesdisplayed on the respective displays as desired (for example by placingthe display more or less toward the center of the field of view of theviewer). As discussed above, the displays may include images fromcameras, images from other sources such as other medical imaging devices(MRI, x-ray, CAT, etc.) or other data. The surgeon may also cause one ormore of the inputs to be scaled up or down or enlarged or reduced to anarea of interest (for example, reduced to a subset of a fluorescent ornarrowband image), which can then be superimposed over another image ormoved with respect to another image by moving one display relative toanother. The beam combiner will combine the images such that they appearsuperimposed or otherwise juxtaposed with respect to each other based onthe relative locations of the respective displays. In this manner, thesurgeon can edit or emphasize particular views, for example, for surgeryas well as edit or emphasize a particular event for teaching ordocumentation purposes.

As described elsewhere in this application and shown, for example, inFIGS. 1 and 11, as well as FIGS. 12A-12C and other figures, the displayassembly need not be a direct view display providing an optical pathfrom the user's eye to the surgical site. Instead, the viewer may viewdisplays that show images captured by cameras viewing the surgical site.Likewise, the view can move about more independently and not be limitedto the location and/or orientation of the camera. Accordingly, theviewer might be 2, 3, 4, 5 or more feet from the surgical site. Thecamera may be less than 1 foot or 2 feet (possibly less than 3 feet)from the surgical site in some such cases.

Cooperative Surgical Display Systems

As part of the surgical visualization systems described herein, asurgical display system can be configured to provide advantageousviewing features to a surgeon, assistant, or other operator of thesurgical visualization system. A surgical display system can beconfigured to provide displays and optical components to view thedisplays. These displays can be configured to provide views of videocaptured with one or more cameras of the surgical visualization system.These displays can also be configured to provide views of otherinformation, such as text data, images, or the like provided by anotherelectronic device. A surgical display system can provide ports, portals,or oculars, transparent plates, lenses, prisms, mirrors, chambers,baffles, and the like to provide optical paths for a viewer to view thedisplays of the surgical display system.

In some embodiments, a surgical display system includes one or moredisplays (e.g., flat panel displays or FPDs) that are configured to beviewed by a surgeon. In the surgical display system, the optical pathsto each eye include a combination of lenses configured to provide a viewalong a right eye optical path and a left eye optical path. Each opticalpath can be directed to the same display (e.g., to different parts of asingle display) or to different displays, for example and withoutlimitation. On the one or more displays, video images can be displayedfrom a collection of sources. Sources can include camera systems of asurgical visualization system. Examples of sources include one or morecameras on endoscopes, one or more cameras providing surgical microscopeviews of a surgical site, one or more proximal cameras mounted on aframe near or adjacent to the surgical site, one or more cameras on asurgical tool, and the like.

In some embodiments, a surgical display system includes one or moredisplays that are viewable in a particular optical path for an eye. Forexample, the right eye optical path can lead to one or more displays andthe left eye optical path can lead to one or more separate displays. Insuch embodiments, each image acquisition camera system can be registeredand aligned to produce a substantially identical view of a scenediffering only in waveband selection. The images can be processed toenhance distinctive features or to differentiate between the imagesprovided in the different wavebands. In some implementations, insertionand registration of pre-surgical information produced by an externalsource can be included on the one or more displays.

In some embodiments, a surgical display system includes one or moredisplays per eye optical path, wherein a display of information or imagein a first display corresponds to a black area in a second display. Thiscan allow the displays to combine to create a seemingly unitary imagewithout ghosting or visual overlapping of different images. The blackportion of the screen can be advantageous as it emits little to no lightto reduce the contrast of the image provided on the first display.

In some embodiments, a surgical display system is provided that includesone or more displays per eye optical path. In such embodiments, a secondimage or portion of a second image can be registered to a first image ona first display. In this way, the first and second images can becombined visually to produce a coherent image.

In some embodiments, the surgical display systems disclosed herein areconfigured so that the right eye path (or right eye view) and the lefteye path (or left eye view) provide different perspectives of the sameimage or images. The surgical display systems can be configured so thatthe exit pupil produced by the system for each eye path is nominallyabout 2-9 mm. The surgical display systems can be configured so that thefield stop produced by the system for each eye is between about 18 mmand about 28 mm. The surgical display systems can be configured so thatthe apparent field of view associated with the exit pupil is betweenabout 45 degrees and about 100 degrees. The surgical display systems canbe configured so that the eye relief for each eye path is nominallybetween about 15 mm and about 30 mm. The surgical display systems can beconfigured so that the total path or track length of the optical pathbetween any display and the viewer's eye (or exit pupil) is at leastabout 50 mm and/or less than or equal to about 400 mm. The surgicaldisplay systems can be configured so that the total track or path lengthof the optical path between any display and the viewer's eye (or exitpupil) contains one or more reflecting surfaces. The surgical displaysystems can be configured so that the total path length of the opticalpath between any display and the viewer's eye (or exit pupil) containsone or more prisms, for example, utilizing total internal reflection.The surgical display systems can be configured so that the one or moredisplays include more than one display per eye path and two or moredisplays are combined with a pellicle or thin glass mounted beamsplitteror combiner coating comprising one or more layers. In such embodiments,a thin glass mounted beamsplitter or combiner can be utilized and theratio of diameter to substrate thickness can be less than about 100to 1. In such embodiments, the beamsplitter or combiner coating can forexample be made of a metalized layer, a patterned layer, or dielectriccoating that has multiple layers.

Binocular Display Unit

Various embodiments include a binocular display unit including a pair ofoculars that produces (for example, for each eye) an exit pupil inspace. This can be a small zone where the marginal rays of the cornersof the field of view cross the optical axis. A person's eye naturallychooses this spot when viewing scenes. At this position, a person sees ablack margin around the field of view. For a modified Wheatstoneconfiguration, the binocular display unit can be configured to utilize arectangular display to take advantage of the natural tendencies of theviewer's vision.

Some embodiments employ a display with an exit pupil (as opposed to alarge eye box). A rectangular field stop can be included in the ocular.In some implementations, rectangular baffles can be included in thedisplay for rejection of stray light. These features can be combinedwith a rectangular panel display for each eye wherein the rectangularpanel displays its half of a stereo image. The display can be positionedat a conjugate position with the field stop in the ocular. A near-eyedisplay places the display at the field stop of the ocular.

The eye relief, e.g., the position of the exit pupil relative to thelast surface of the ocular, is a factor that determines how close aviewer is to the device when viewing the displays. When the eye reliefis greater than about 15 mm to about 20 mm, the observer can comfortablywear spectacles during use of the device.

The oculars of the binocular display unit can be used to provide a viewwith a larger apparent field of view. The ocular can be configured toprovide the power and magnification to produce an exit pupil at adesired or targeted location thereby providing desired or targeted eyerelief from a field stop. This can produce this apparent field of view.

Beyond a certain field of view (where field of view is inverselyproportional to the focal length of the ocular), the output of theocular does not generally couple with the pupil of an eye and a viewertypically fails to see part of the field. This effect can be referred toas a kidney bean effect, as the portion not seen has a kidney beanshape. A 10× magnification operating room microscope ocular, forexample, has a wide field of view and comfortable exit pupil position.In the binocular display unit, the last components of the display system(e.g., the oculars), enable the apparent field of view.

Another consideration when designing a display unit that uses an eye boxrather than an exit pupil is illumination. As the apparent field of viewincreases, the relative brightness of the display decreases. So, as theexit pupil grows until it becomes an eye box, the display panelsdecrease in apparent brightness because the area (of the eye box) islarger. Accordingly, the disclosed binocular display units put asignificant amount of the total illumination into a useful sized exitpupil (as opposed to an eye box). This facilitates interoperability andintegration into a surgical environment as the exit pupil of operatingroom microscopes is a well-established and accepted size.

Various embodiments of the binocular display unit provide a compact unitwith ergonomic adjustments at or near the oculars. These display unitscan be configured to be less massive than other display units meaningthat it can be easier to move and adjust.

In some embodiments, a display unit can include a display housing, anopening in the display housing, at least two chambers within the displayhousing (one for the left eye and one for the right eye), and at leastone electronic display disposed the display housing, each of the atleast one electronic display comprising a plurality of pixels configuredto produce a two-dimensional image. The separate left and right opticalpaths together provide stereo viewing. The display housing is furtherconfigured to separate a right eye path from a left eye path to the atleast one electronic display so that light intended to be viewed with aright eye from the at least one electronic display does not travel downthe left eye path and vice versa. The medical apparatus includes animaging system disposed on the display housing, the imaging systemconfigured to generate images of a surgical site from outside thesurgical site. The view of the display within the housing can beconfigured to provide a stereoscopic image to a viewer, as discussedabove. In some embodiments, the housing further includes lenses and/or atransparent plate between the eyes of a viewer and the at least oneelectronic display. In some embodiments, the housing further includes abaffle or other structure to separate the left eye path and the righteye path. In some embodiments, each chamber is baffled to prevent lightfrom one channel to communicate to the other eye path. In someembodiments, at least a portion of the chambers comprise oculars.

In certain implementations, the display unit can be baffled to preventlight communication between the left and right eye channels. To adjustfor different accommodations, the displays within the display system canbe configured to move toward and/or away from the viewer along theoptical path. This can have an effect similar to varying focal lengthsof lenses in an ocular system. In some embodiments, the display housingwith the electronic displays can change to move the displays closer orfurther from the viewer along the optical path. In some embodiments,both the display housing and the electronic displays are configured tobe adjustable along the optical path to adjust for accommodation.

In certain implementations, the binocular display unit can be configuredto receive video images acquired with an endoscope and display thesevideo images within the unit. These images can be combined (e.g.,stitched, tiled, switched, etc.) with video from other sources by anoperator so that video images from one or more sources, including theendoscope video images, can be viewed with the binocular display unit.

Display Unit with Folded Optical Path

Various embodiments include a display unit including optical elementsconfigured to redirect an optical path from a viewer's eye to a displaysuch that the display is positioned above the viewer's eye when theviewer is looking substantially horizontally (e.g., where horizontal isperpendicular to gravity which is vertical) into the display unit. FIG.13B1-a illustrates an example of such a configuration for a display unitwherein optical components direct and focus images from displays 3005 aand 3006 a to the viewer using a beamsplitter/combiner 3004 and mirroror reflector 3007. This allows the displays 3005 a and 3006 a to bepositioned above the level of the eye when the viewer is lookingsubstantially horizontally (e.g., with gravity vertical) into thedisplay unit. This may also be the case where the majority or all of theoptical components are not above the level of the eye when the viewer islooking substantially horizontally into the display unit.

In some embodiments, the display unit includes oculars through which theviewer looks to view images provided by the displays 3005 a, 3006 a. Insome embodiments, the display unit includes eye portals through whichthe viewer looks to view images provided by the displays 3005 a, 3006 a.In either of these embodiments, the oculars or eye portals and thedisplay unit can include optical components configured to direct theoptical path from the displays to the oculars or eye portals as well asto provide imaging functionality. In some embodiments, dimensions of thedisplay housing include the oculars or eye portals and in someembodiments, dimensions of the display housing exclude the oculars oreye portals.

To facilitate the description of the configuration of the display unit,a coordinate system will be adopted that is fixed to the display unit.The coordinate system will take the optical axis through the oculars oreye portals as the x-axis (an example of which is illustrated in FIGS.13B1-e and 13B1-f) and the optical axis after the redirection opticalcomponent (e.g., the mirror or reflector 3007 in FIG. 13B1-a) as they-axis (an example of which is illustrated in FIGS. 13B1-e and 13B1-f),the x-axis being perpendicular to the y-axis. The physical distancealong the x-axis from the display to the oculars can be a fraction ofthe distance along the y-axis from the display to the oculars. Forexample, the distance along the x-axis to the display can be less thanor equal to 90% of the distance along the y-axis to the display, lessthan or equal to 80% of the distance along the y-axis to the display,less than or equal to 70% of the distance along the y-axis to thedisplay, less than or equal to 60% of the distance along the y-axis tothe display, or less than or equal to 50% of the distance along they-axis to the display or any ranges therebetween. In some embodiments,after the optics of the oculars or eye portals, the majority of theoptical components can be positioned above (e.g., in a positivedirection along the y-axis as defined) the oculars or eye portals. Insome embodiments, a majority of the length of the optical path can bepositioned above the oculars or eye portals. For example, the totallength of the optical path can be taken as the distance from a firstoptical component (e.g., window where a window may be a lens or planartransparent plate) of the oculars or eye portal to the display 3005 a or3006 a. In some embodiments described herein, less than 50% of thattotal length is the length of the optical path before the mirror 3007 orthe optical component that redirects the optical path from beingsubstantially along the x-axis to being substantially along the y-axis.In some embodiments, the optical axis does not travel downward along they-axis. In some embodiments, the optical axis does not travel downwardalong the y-axis prior to being redirected upward along the y-axistoward the displays. These features can allow for a display unit that islarger in height than in depth. This may be advantageous to allow for adisplay unit that can be moved around in an operating room environmentwithout substantially interfering with the positioning of other displayunits, camera assemblies, and/or personnel.

To further describe the configuration of the display unit, thecoordinate system defined above can include a plane that isperpendicular to the y-axis, referred to as the viewing plane, where theviewing plane intersects the optical axis at the oculars or eye portals.In some embodiments, the display unit can be configured so that amajority of the optical path is not below this viewing plane. In someembodiments, the display unit can be configured so that a majority ofthe optical path is above this viewing plane. In some embodiments, thedisplay unit can be configured so that at least 40%, at least 50%, atleast 60%, at least 70%, at least 80%, or at least 90% of the opticalpath is above the viewing plane or any value therebetween. In someembodiments, the display unit can be configured so that at least 40%and/or less than or equal to about 99%, at least 50% and/or less than orequal to about 95%, at least 60% and/or less than or equal to about 90%,or at least 70% and/or less than or equal to about 85% of the opticalpath is above the viewing plane. In some embodiments, the display unitcan be configured so that a majority of the optical components are abovethe viewing plane. In some embodiments, the display unit can beconfigured so that the displays are above the viewing plane. In someembodiments, the display unit can be configured so that a majority ofthe display housing volume is above the viewing plane. In someembodiments, the display unit can be configured so that at least 40%, atleast 50%, at least 60%, at least 70%, at least 80%, or at least 90% ofthe display housing volume is above the viewing plane. In someembodiments, the display unit can be configured so that at least 40%and/or less than or equal to about 99%, at least 50% and/or less than orequal to about 95%, at least 60% and/or less than or equal to about 90%,or at least 70% and/or less than or equal to about 85% of the displayhousing volume is above the viewing plane.

In some embodiments, the display units can be positioned so that theirdisplay panels are substantially perpendicular to one another, asillustrated in FIG. 13B1-e. In some embodiments, the display units canbe positioned so that their display panels are substantially parallel toone another, as illustrated in FIG. 13B1-f. In such an embodiment, anadditional mirror 3004 b can be included above the beamsplitter/combiner3004 to redirect the optical axis to the other display.

The display unit can have a depth, d, (e.g., the extent of the displayunit housing along the x-axis) and a height, h, (e.g., the extent of thedisplay unit housing along the y-axis) that forms an aspect ratio (d:h)that is less than 1. The aspect ratio of the depth to height can be lessthan or equal to about 0.95, 0.9, 0.85, 0.8, 0.75, 0.7, 0.65, 0.6, 0.55,0.5, 0.45, or 0.4 or ranges therebetween any of these values (e.g.,0.95-0.6 or 0.9-0.65).

To facilitate the description of other configurations of the displayunit, another coordinate system will be adopted that is fixed relativeto gravity. The coordinate system will take gravity as the y-axis andthe x-axis lies within the horizontal plane perpendicular to gravity,the x-axis being the projection of the optical axis at the exit window(e.g., final lens element of the oculars or eye portal) on thehorizontal plane. In this coordinate system, the height of the displayunit can be taken as the dimension of the display housing along they-axis (e.g., the direction parallel to gravity) and the depth can bethe dimension taken along the x-axis (e.g., the direction perpendicularto gravity and parallel to the projected optical path at the ocular oreye portal). With this convention, the display unit can have a depth, d,(e.g., the extent of the display unit housing along the x-axis) and aheight, h, (e.g., the extent of the display unit housing along they-axis) that forms an aspect ratio (d:h) that is less than 1. The aspectratio of the depth to height can be less than or equal to about 0.95,0.9, 0.85, 0.8, 0.75, 0.7, 0.65, 0.6, 0.55, 0.5, 0.45, or 0.4 or rangestherebetween any of these values (e.g., 0.95-0.6 or 0.9-0.65). In someimplementations, the optical path of the oculars or eye portals can forman angle that is less than or equal to about 30 degrees with thehorizontal plane, less than or equal to about 15 degrees with thehorizontal plane, less than or equal to about 10 degrees with thehorizontal plane, less than or equal to about 5 degrees with thehorizontal plane, or less than or equal to about 0 degrees with thehorizontal plane.

In some embodiments, the display housing can have a height that islarger than its depth. For example, the depth can be less than or equalto about 90% of the height, less than or equal to about 80% of theheight, less than or equal to about 70% of the height, less than orequal to about 60% of the height, less than or equal to about 50% of theheight, or less than or equal to about 40% of the height. In someembodiments, the display housing can be longer than it is deep. Forexample, the display housing can be at least 10% longer than it is deep,at least 25% longer than it is deep, at least 50% longer than it isdeep, or at least 100% longer than it is deep. In some implementations,the optical path within the display housing can be longer along itsheight than along its depth. For example, the optical path along theheight can be at least 10% longer than along its depth, at least 25%longer than along its depth, at least 50% longer than along its depth,or at least 100% longer than along its depth or any range of valuestherebetween.

In some embodiments, the display unit can include an optical system(e.g., lenses) with one or more two-dimensional displays andillumination sources, the optical system configured to direct rays fromthe displays and illumination sources towards the eyes of a viewer, theoptical system configured to produce a collimated virtual image of theone or more displays for each eye. A majority or all of the opticalsystem of the display unit can be near or above a plane that is parallelto the viewer's line of site when looking through oculars or eye portalsinto the display unit to see the displays. In certain implementations,the collimated rays can be disposed at an angle relative to a verticalplane that is substantially perpendicular to the plane parallel to theviewer's line of site. In some implementations, the display unitincludes one or more fold mirrors or prisms or reflectors that arepositioned between the displays and the optical system enabling the useof displays that are wider than the center-to-center spacing of theocular or eye portals or the viewer's eyes, an example of which isillustrated in FIGS. 13B1 and 13B1-b. In some implementations, the oneor more fold mirrors or prisms or reflectors lie within the opticalsystem rather than between the optical system and the displays. In someimplementations, the optical system includes one or more fold mirrors orprisms or reflectors configured to permit displacement (e.g., in thez-direction) of each eye's opto-mechanical axis to permit a compactoverall design and ergonomic adjustment. In some implementations, one ormore fold mirrors or prisms or reflectors are positioned between thedisplays and the optical system, the one or more fold mirrors or prismsconfigured to permit displacement (e.g., in the z-direction) of eacheye's opto-mechanical axis to permit a compact overall design andergonomic adjustment.

Accommodation Differences

In some embodiments, a display unit can include a display housing, anopening in the display housing, at least two chambers within the displayhousing, and at least one electronic display disposed in the displayhousing, each of the at least one electronic display comprising aplurality of pixels configured to produce a two-dimensional image. Thedisplay housing is further configured to separate a right eye path froma left eye path to the at least one electronic display so that lightintended to be viewed with a right eye from the at least one electronicdisplay does not travel down the left eye path and vice versa. Themedical apparatus includes an imaging system disposed on the displayhousing, the imaging system configured to generate images of a surgicalsite from outside the surgical site. The view of the display within thehousing can be configured to provide a stereoscopic image to a viewer.In some embodiments, the housing further includes lenses and/or atransparent plate between the eyes of a viewer and the at least oneelectronic display. In some embodiments, the housing further includes abaffle or other structure to separate the left eye path and the righteye path. In some embodiments, each chamber is baffled to prevent lightfrom one channel to communicate to the other eye path. In someembodiments, at least a portion of the chambers comprise oculars.

In certain implementations, the display unit can be baffled to preventlight communication between the left and right eye channels. To adjustfor different accommodations, the displays within the display system canbe configured to move toward and/or away from the viewer along theoptical path. This can have an effect similar to varying focal lengthsof lenses in an ocular system. Similarly, lenses or other opticalcomponents can be moved in the system to provide for differentaccommodations. In some embodiments, accommodation adjustment can bemade by moving the screens relative to the lenses inside the displayunit. In certain implementations, by lengthening the housing with thescreens attached, or by moving the screens within a fixed housingrelative to the lenses and user, or by moving the position and/or theseparation of lenses or groups of lenses, accommodation can be adjusted.Additional adjustments between the two chambers can be accomplished byphysically moving the two chambers laterally, e.g., by separating ormoving the chambers closer together to adjust for inter pupillarydistance differences between users.

In some embodiments, the display housing with the electronic displayscan change to move the displays closer or further from the viewer alongthe optical path. In some embodiments, both the display housing and theelectronic displays are configured to be adjustable along the opticalpath to adjust for accommodation.

In certain implementations, the binocular display unit can be configuredto receive video images acquired with an endoscope and display thesevideo images within the unit. These images can be combined (e.g.,stitched, tiled, switched, etc.) with video from other sources by anoperator so that video images from one or more sources, including theendoscope video images, can be viewed with the binocular display unit.

Camera Providing Surgical Microscope Views

The beginning, middle and end of a surgical case may involve differingvisualization goals. As the case begins, the surgeon may be interestedin surveying and viewing the area for the surgery. At skin level thesurgeon may use surgical tools and the operating room microscope toguide their progress into the body to the surgical site. This means whatis desirable is an image acquisition system functioning like anoperating room microscope (OR scope) until such time as they choose touse views from other cameras (e.g., one or more cameras on a surgicaltool(s), endoscope(s), one or more proximal cameras, etc.).

With this display design a number of surgical microscope camerafunctions are enabled. As described herein, the difference between usingan exit pupil versus using an eye box results in differing apparentfields of view. Similarly, the choice of an electronic acquisitionsystem that is not optically connected to the displays, as is the casein typical OR microscopy, makes many features possible.

An optically coupled microscope may have huge arms and complicatedmotions to position the microscope in positions required for surgery, inparticular for neurosurgery. Some stands give the operating roommicroscopes a wide range of counterbalanced motions. Decoupling theoptical system acquisition from the display, allows for a far greaterrange of motion, and this motion is closer to the optical elements ofthe system rather than the stand. Such a decoupled optical system canenable a compact system that can eliminate most or all counterbalancingefforts due at least in part to the reduced size and mass. Similarly,decoupling image acquisition of the surgical microscope view andproviding an electronic display with no direct optical path from theoculars to the surgical site affords the opportunity to make a compactand ergonomic system.

Oblique surgical microscope views are useful, for example, forneurosurgery. One challenge to overcome in such systems is if there is arotation of the surgical microscope views for oblique views there may bea roll component with the following result. The surgeon's eyes are in aplane parallel with the horizon and parallel with the display. Adding anoblique view in the right eye and left eye path's rolling, meaning theright eye and left eye may rock up and down with respect to one anotheras the mechanism is repositioned.

If the pitch is in a line coincident with the primary surgeon's gaze,from vertical to oblique, yaw around a central axis position-able in xand y, a collar to rotate surgical microscope cameras to switch viewsbetween the pairs of optics or alternatively the upper gimbal can berotated 90 degrees and the view through the right eye left eye pairs canbe switched electronically to give a roll motion in a vertical position.The pitch and yaw and collar gimbals are can be motor driven controlledby a joystick on one of the handles that manually control x and y of theassembly. A fine focus z adjustment can be manual or motor driven fromcontrols or mechanisms on one handle or on one of the 2 handles oneither side of the assembly. Zoom functions, illumination controls,fixing gaze position etc. can be controlled from the handles as well.The one handle or the 2 handles can reposition the entire mechanismunder the display in x and y without disturbing the position of thedisplay.

This assembly does not introduce roll in an oblique view, but could bepositioned in a vertical position to have some roll if that is desired.For example, if one wants to roll the view slightly to one side oranother so one or the other eye's view is not obstructed by tool use inthe surgical opening when the surgical microscope camera is used in asubstantially vertical position.

In some embodiments, an electronic surgical microscope for one or moresurgeons to view a stereo pair of images in an electronic display from asurgical site is provided that includes a right eye path and a left eyepath through a common objective. The electronic surgical microscope canbe configured to be used at the focal length of the objective.

In a further embodiment, the electronic surgical microscope can includea right eye path comprising a view through a common objective and anafocal zoom assembly as well as a left eye path comprising a viewthrough a common objective and an afocal zoom assembly.

In a further embodiment, the electronic surgical microscope can includea right eye path comprising a view through a common objective and anafocal zoom assembly and an adjustable diaphragm as well as a left eyepath comprising a view through a common objective and an afocal zoomassembly and an adjustable diaphragm.

In a further embodiment, the electronic surgical microscope can includea right eye path comprising a view through a common objective and anafocal zoom assembly and an adjustable diaphragm and a focusing lensassembly as well as a left eye path comprising a view through a commonobjective and an afocal zoom assembly and an adjustable diaphragm and afocusing lens assembly.

In a further embodiment, the electronic surgical microscope can includea right eye path comprising a view through a common objective and anafocal zoom assembly and an adjustable diaphragm and a focusing lensassembly forming an image on a detector. In a further embodiment, theelectronic surgical microscope can include a left eye path comprising aview through a common objective and an afocal zoom assembly and anadjustable diaphragm and a focusing lens assembly forming an image on adetector.

In a further embodiment, the electronic surgical microscope can include2 or more real time video camera systems coupled to the right eye andleft eye paths for processing signals from the right eye and left eyedetectors, wherein electronic signals of each eye path produceresolution compatible with HD displays, e.g. 720i, 720p, 1080i, 1080p,and 4 k.

In a further embodiment, the electronic surgical microscope can beconfigured so that neither eye path produces an aerial image suitablefor direct viewing.

In a further embodiment, the electronic surgical microscope can includea system to divide the output of each detector, right and left eye, andvertically flip the images to a second right eye and left eye displayfor an assistant surgeon at 180 degrees to the primary surgeon.

In a further embodiment, the electronic surgical microscope can includea second pair of stereo paths that include a right eye path comprising aview through a common objective and an afocal zoom assembly and anadjustable diaphragm and a focusing lens assembly forming an image on adetector, a left eye path comprising of a view through a commonobjective and an afocal zoom assembly and an adjustable diaphragm and afocusing lens assembly forming an image on a detector, where both stereopaths are rotated 90 degrees from the first stereo pair paths to permita second surgeon to view through the common objective and sit at 90degrees to the primary surgeon. In addition, it can be configured sothat neither eye path produces an aerial image suitable for directviewing. In yet a further embodiment, the right eye path and the lefteye path to their respective detectors permit an assistant surgeon tosit at right angles to the primary surgeon. Similarly, the output of theright eye and left eye detector of the assistant surgeon's detector maybe vertically flipped to display the stereo scenes appropriately whetherthe assistant surgeon sits on the right or left side of the primarysurgeon. Likewise, the surgical microscope can include a collar tosupport the stereomicroscope permitting the 4 eye paths to be rotatedaround the line of sight within the collar plus or minus 90 degrees ormore and a system to switch stereo pairs displayed in the viewer fromassistant surgeon to primary surgeon.

In some embodiments, a surgical microscope image acquisition system canbe provided for acquiring stereo images of a surgical site to bedisplayed for one or more surgeons without a direct path opticallybetween the lens elements of the acquiring system and surgeon's eyes. Ina further embodiment, the surgical microscope image acquisition systemcan be suspended below an electronic display system for surgery andattached to a plane parallel with the horizon. The plane is the bottomsurface of an electronic display for viewing the output of thestereomicroscope image acquisition system. In some embodiments, theplane is integral with a proximal column providing rotation, yaw, forintegral displays and one or more stereomicroscope image acquisitionsystems. In a further embodiment, the column attaches to an arm whichprovides x y and z positioning movement for entire acquisition anddisplay system at, over or adjacent to the patient. In a furtherembodiment, the arm attaches to a vertical column supporting all of theabove which provides z, rotation and yaw. In some embodiments, thecolumn can provide gross positioning and focusing without the need ofcounter balance measures as seen in operating room microscope stands,due to the lack of a directly optical pathway from the surgical site tothe doctor's eyes.

Decoupling image acquisition from display in a Wheatstone like stereodisplay results in less mass and less movement of the display(s) thatcan result in an ergonomic viewing position for long cases. The primarypositioning of a surgical view can be by the gimbaled image acquisitionsystem. The positioning elements can be moved from adjacent to column incompeting solutions to adjacent to the image acquisition systemdescribed herein that provides surgical microscope views.

In some embodiments, a surgical microscope image acquisition systemwhose stereo pair eye paths view a surgical site through a commonobjective contains one or more zoom (or just ‘lens’) assemblies totransform the numerical aperture of one or more fiber optic cables froma remote source(s) also functioning through a common objective. By thismeans the divergence of the illumination light may be altered toilluminate a scene as either the imaging zoom changes, or to place moreenergy in an area.

In some embodiments, a surgical microscope image acquisition system isprovided where the fiber optic illumination zoom functioning through acommon objective has a gimbal mechanism so that the illumination may besteered within the surgical opening, or on the patient.

Various embodiments include a gimbaled image acquisition system, whichhas a working distance range and stereo separation of an operating roommicroscope, without an optical through path to the surgeon's eyes. Theacquisition system is mounted in a collar (described above) in a gimbalsystem so that four zoom paths can be rotated 90 degrees around the lineof sight to switch stereo paths between assistant surgeon and primarysurgeon.

The acquisition system and collar attach to a yoke, which allows thesurgical microscope image acquisition system to pitch, e.g., view closerto or further away from the surgeon.

An electronic stereomicroscope for one or more surgeons is provided toview a stereo pair of images in an electronic display from a surgicalsite having separate right eye and left eye paths (e.g., a Greenoughconfiguration) with a variable convergence. In some embodiments, theelectronic stereomicroscope can include a right eye path comprisingtilted afocal zoom assembly providing variable convergence angle optionas well as a left eye path comprising tilted afocal zoom assemblyproviding variable convergence angle option.

FIGS. 8A-8B illustrate views of a camera providing a surgical microscopeview rotated around a central axis from the point of view of the primarysurgeon. The x y stage under the display allows the surgical camera viewmechanism to be shifted from one side to another giving the surgeonbetter access to the surgical site for tool use. The assistant scope canbe moved from one side of the device to the other, e.g., +/−90 degrees,from the position of the primary surgeon. The gimbal on the yoke allowsthe device to see retrograde from vertical. The gimbal system can bepositioned for an oblique side view. The device can be configured foroblique views which may be particularly advantageous for neurosurgery.

Frame and Proximal Cameras

Certain embodiments include a medical apparatus comprising one or moreproximal and/or distal cameras. In some cases (e.g., in some cases ofbrain surgery), minimal retraction and/or momentary retraction may bedesired. Accordingly, certain embodiments of a medical apparatuscomprising a frame, which is not a retractor, can be beneficial.Accordingly, various embodiments of a medical apparatus can comprise aframe that is not a retractor. The frame may be configured to bedisposed above a surgical site of a patient. The frame can be mounted toa bed or to the patient and in some embodiments, anchored outside thesurgical site of the patient. The medical apparatus can also include oneor more cameras (e.g., a stereo camera, a mono camera, a cameraproviding a surgical microscope view, etc.) mounted to the frame. Theone or more cameras can be configured to image the surgical site.

FIG. 14A shows a schematic of an example of such a medical apparatus 500comprising a frame 510 disposed above a surgical site 501 of mockpatient. The mock patient includes an opening 505 in the skull 507,material 509 representing brain, and material 511 representing whitetissue. A hand retractor 513 is shown as lifting the material 509representing brain to show the skull base tissue 511. The frame 510 canbe configured to be disposed above the surgical site 501 of the patient.One or more cameras 520 can be mounted to the frame 510. For example,referring to FIG. 14A, one or more cameras 520 can be mounted to theframe 510 via mounts, clamps, and/or fingers 525 (with or withoutgimbals 530).

In various embodiments, the frame 510 can be configured to be mounted toabed (e.g., to a gurney) or to the patient. For example, the frame 510can be configured to be mounted to the bed (e.g., to the bed rail)and/or to the patient via a Mayfield clamp or a Mayfield mount. Theframe 510 can be configured to be disposed outside the patient butwithin a close proximity to the patient and/or surgical site 501. Forexample, the frame 510 can be configured to be disposed above thesurgical site 501 and/or above the patient by a distance of 1 mm, 5 mm,10 mm, 15 mm, 20 mm, 25 mm, 30 mm, 35 mm, 40 mm, 45 mm, 50 mm, 75 mm,100 mm, 120 mm, 130 mm, 140 mm, 150 mm, 175 mm, 200 mm, 250 mm, 300 mmor any value in between these values. Accordingly, in variousembodiments, the frame 510 can be configured to be disposed, forexample, 1 mm to 50 mm, 5 mm to 40 mm, 5 mm to 50 mm, 10 mm to 25 mm, 10mm to 40 mm, 10 mm to 50 mm, 50 mm to 75 mm, 50 mm to 100 mm, 50 mm to120 mm, 50 mm to 130 mm, 50 mm to 140 mm, 50 mm to 150 mm, 100 mm to 200mm, (or any range formed by any of the values from 1 mm to 300 mm) abovethe surgical site 501 and/or above the patient. In various embodiments,the camera location can be configured to be disposed above the surgicalsite 501 and/or above the patient by a distance of 1 mm, 5 mm, 10 mm, 15mm, 20 mm, 25 mm, 30 mm, 35 mm, 40 mm, 45 mm, 50 mm, 75 mm, 100 mm, 120mm, 130 mm, 140 mm, 150 mm, 175 mm, 200 mm, 250 mm, 300 mm or any valuein between these values. Accordingly, in various embodiments, the frame510 can be configured to be disposed, for example, 1 mm to 50 mm, 5 mmto 40 mm, 5 mm to 50 mm, 10 mm to 25 mm, 10 mm to 40 mm, 10 mm to 50 mm,50 mm to 75 mm, 50 mm to 100 mm, 50 mm to 120 mm, 50 mm to 130 mm, 50 mmto 140 mm, 50 mm to 150 mm, 100 mm to 200 mm, (or any range formed byany of the values from 1 mm to 300 mm) above the surgical site 501and/or above the patient. In some embodiments, the camera can extendclose to and/or inside the surgical site even if the frame 510 islocated above the surgical site.

The one or more cameras 520 can include any of the cameras describedherein. For example, the one or more cameras 520 can include a mono viewcamera, e.g., a single camera with one field of view with a circular,rectangular, or square image output. In various embodiments, the one ormore cameras 520 can provide a left-eye view and a right-eye view. Insome such embodiments, the one or more cameras 520 can be configured toprovide stereo imaging. For example, the one or more cameras 520 caninclude a stereo view camera, stereo assemblies, a pair of cameras, or asplit sensor with one-half providing the right-eye view and the otherhalf providing the left-eye view. The one or more cameras 520 caninclude one or more illumination sources.

Various embodiments of the medical apparatus 500 include a frame 510 forthe purpose, partially or predominantly, of holding one or more cameras520 (e.g., one or more cameras as described herein). Because the frame510 is not a retractor and does not move or retract tissue, a handretractor 513 can be used to move tissue in the surgical site 501. Invarious embodiments, one or more cameras 520 can be configured to bemounted to the frame 510. In some embodiments, the one or more cameras520 can face inwardly and/or downwardly into the surgical site. The sizeof the frame 510 is not particularly limited and can depend on the sizeof the surgical site 501 and/or the type of surgical procedure. Theshape of the frame 510 is also not particularly limited. As someexamples, the frame 510 can have a cross-sectional shape comprising around shape (e.g., a circle, an oval, etc.), a regular polygon (e.g., asquare, a rectangle, a hexagon, an octagon, etc.), or an irregularpolygon (e.g., an L-type shape). FIGS. 14A1-a, 14A1-b, and 14A1-cschematically illustrate an example circular frame 510 a, an examplesquare frame 510 b, and an example L-shaped frame 510 c respectively.

Various embodiments of the medical apparatus 500 can include one or morecameras 520 mounted to the frame 510 to provide one or more perspectivesof the surgical site 501. In some embodiments, proximal cameras can bemounted to the frame 510 providing at least two or at least fourdifferent perspectives. As one example, four cameras 520 can be mountedto the frame 510 at 3 o'clock, 6 o'clock, 9 o'clock, and 12 o'clockpositions. As shown in FIGS. 14A1-a, cameras 520 a are mounted to frame510 a at 3, 6, 9, and 12 o'clock positions. In addition, as shown inFIGS. 14A1-b, cameras 520 b are mounted to frame 510 b at 3, 6, 9, and12 o'clock positions. In various embodiments, referring to FIG. 14A, thecameras 520 can be positioned 90 degrees (or other angles) apart fromeach other. Accordingly, referring to FIG. 14A, the cameras 520 can bepositionable around the frame 510, for example, around a ring or arounda square. In some instances, the cameras 520 can be symmetricallypositioned around the frame 510. In other instances, the cameras 520 canbe asymmetrically positioned around the frame 510. As additionalexamples, cameras 520 can be mounted to an L shape of one kind oranother. In FIGS. 14A1-c, cameras 520 c are mounted to the L-shapedframe 510 c.

By being mounted to the frame 510, the one or more cameras 520 can imagethe surgical site 501. Certain embodiments described herein can includeone or more mounts (or clamps or fingers) 525 connecting the one or morecameras 520 to the frame 510. Although the cross-sectional shape of themounts 525 in FIG. 14A is schematically shown as rectangular, the shapeof one or more mounts 525 is not particularly limited.

Various embodiments of the medical apparatus 500 can include one or moregimbals 530 and/or movement control systems (e.g., other positioningand/or orientation systems similar to any of those described herein) tocouple a camera 520 to the frame 510 and to change the positioningand/or orientation of the camera 520. In some embodiments, the one ormore gimbals 530 can provide gimballing motions similar to thosedescribed for cameras providing a surgical microscope view underneath adisplay. For example, various embodiments utilize a gimbal systemsimilar to those from under the camera providing a surgical microscopeview, yet can be relatively smaller and disposed on mounts, clamps, orfingers 525. As will be described herein, various embodiments caninclude attachment points 515 for four-bar or other x-y-z mechanisms(see, e.g., FIGS. 14A1-a, 14A1-b, and 14A1-c).

Accordingly, in various embodiments, one or more cameras 520 can includefirst and second cameras configured to move relative to the surgicalsite 501. For example, certain embodiments of the medical apparatus 500include one or more gimbals 530 mounted to one or more mounts 525. Theone or more gimbals 530 can be configured to allow positioning of one ormore proximal cameras 520, such as those not inside the surgical site501 but viewing into it. Such embodiments can have some advantageouspositions. Thus, in various embodiments, one or more gimbals 530 can beconfigured to move the one or more cameras 520 relative to the frame510. Some embodiments can include knobs or heads to turn one or morecameras 520 in and out. Some embodiments can also include some featuresto drive one or more cameras 520 in a precise manner (e.g.,sub-millimeter range of motions). In some embodiments, three, four, ormore knobs can be provided on either side on the external side (e.g.,the outboard side) of the frame 510. By turning the knobs, one or moregimbals 530 can be oriented to move in one axis or the other axis, ormove in an x axis or move in a y axis, or move in a z axis, which can bea very advantageous situation. In some embodiments, one or more cameras520 can be configured to move electronically.

As described herein, the one or more cameras 520 can include any of thecameras described herein. For example, the one or more cameras 520 caninclude a stereo view camera, stereo assemblies, a pair of cameras, or asplit sensor with one-half providing the right-eye view and the otherhalf providing the left-eye view.

One of the advantages of stereo imaging is the ability to provide greatdepth perception. One of the downsides of any kind of surgery inside thebody is becoming disoriented. To give an orientation of location insidethe body, surgeons can look at surgical landmarks, the direction of alight source, an external part of an endoscope (e.g., outside the body),and/or a cable coming out of the scope. Additionally, an anatomicallandmark relative to others can also be used. However, tissue with verydifferent functions can look very similar, thus resulting indisorientation. Stereo imaging, while it can give great depthperception, can actually add to confusion. If surgeons can be certainthat they are always in a horizontal orientation, it can be an extremelyadvantageous feature, because at least surgeons can know they were notrotated. For example, surgeons may still have some orientation issueabout location. However, knowing that the right eye and left eye areseeing things as though standing on the floor and upright, with theright eye and left eye in the same plane, can be extremely advantageousto avoid becoming disoriented.

Utilizing one or more gimbals 530 (and/or other positioning and/ororientation systems), e.g., including those having six degrees offreedom, can allow the horizontal line of sight, e.g., right eye andleft eye, to have the same orientation as the viewer. For example,certain embodiments can maintain the horizontal in a way that does notallow any other motion other than keeping the right eye and left eyehorizontal orientation, whether looking straight down or in an obliquesituation. As described herein, this can be advantageous in someinstances. If a surgeon has an obstruction, it may be desired to anglethe surgeon's view to see around the distraction, but to maintain ahorizontal line of sight or at least not to introduce roll.

Accordingly, in certain embodiments, the one or more cameras 520 caninclude first and second cameras configured to move relative to thesurgical site 501 and to maintain a same horizontal orientation withrespect to each other. For example, referring to FIG. 14A, the mounts525 can hold the six degrees of freedom (or however many degrees offreedom desired) to create a gimbal that moves around (e.g., to allow agimbal 530 to move around) but, can in various embodiments, maintain thehorizontal orientation of right eye and left eye in the same plane orallow the viewer to not feel the right eye or left eye higher than orlower than the other eye. With continued reference to FIG. 14A, incertain embodiments, a camera 520 can be relatively smaller than themount 525. The gimbal 530 can have a relatively small pitch yaw rotationstructure that can move the camera 520 to direct its line of sight insix degrees of freedom. Certain embodiments provide x-y-z and pitch,yaw, and roll mainly both to position the camera 520 but also tomaintain the horizontal axis of the two eye views. For example, invarious embodiments, the one or more gimbals 530 can be configured topotentially allow for repositioning in (transverse) x, y, or(longitudinal) z direction as well as pitch or yaw (rotation), or anycombination thereof. Various embodiments, however, might not enable rollso as to reduce the disorientation that may result when the left andright channels of a stereo camera are rolled (e.g., the horizontal linethrough the left and right channels are not parallel to the ground).Accordingly, the one or more cameras 520 can be configured to move withrespect to an x direction, a y direction, or a z direction. In someembodiments, one or more cameras 520 can be configured to move withrespect to a pitch or yaw. For example, one or more cameras 520 can beconfigured to move with respect to a pitch and/or yaw, and without roll.

As described herein, in various embodiments, the frame 510 can beconfigured to be mounted (directly or indirectly) to a bed (e.g., agurney) and/or to the patient and anchored outside the surgical site ofthe patient. In some embodiments, the frame 510 can be configured to bemounted to the bed (e.g., to the bed rail) and/or to the patient via aMayfield clamp or a Mayfield mount. In some embodiments, the frame 510can be configured to provide a stereotactic planning system. Forexample, a system that provides a frame of reference, e.g., a coordinatesystem associated with many positions in that region can be used.Certain embodiments can be drilled right into or otherwise attached to apatient (e.g., skull for brain surgery). If the patient moves (e.g.,coughs), the whole device moves. Accordingly, in certain embodiments,the frame of reference may not get disoriented by involuntary orvoluntary movement of the patient. In various embodiments, the frame 510can be a stereotactic frame. In various embodiments, the frame 510 canbe mounted to a stereotactic system. In various embodiments, the frame510 can be mounted to the patient and connected indirectly to astereotactic system.

Certain embodiments of the medical apparatus 500 can be supported by thebed, by a Mayfield clamp, by a Mayfield mount, or by the frame 510(e.g., a stereotactic frame). For example, in certain embodiments, astereotactic frame or the bed provides a supporting structure. In someembodiments, a stereotactic frame can mount on the patient's skull, andsome other support mechanism can mount to the bed.

Certain embodiments of the medical apparatus 500 can include more distalcameras than the proximal cameras 520. For example, a more distal camerathan the proximal camera 520 can be mounted to the frame 510, e.g., on afinger 525 that goes down into the surgical site 501. The cameras 520can be mounted to face inward with respect to each other (and/orpossibly downward into the surgical site). Various embodiments includingcameras 520 on a frame 510, such as a stereotactic frame or abed-mounted frame, can include features applicable to other technologyincluding but not limited to distal and proximal cameras.

In certain embodiments of proximal cameras 520 on frames 510, theproximal cameras 520 can be positioned just outside and adjacent to thesurgical site 501 (e.g., between 5 mm and 50 mm, between 10 mm and 25 mmabove the patient, between 10 and 40 mm above the patient, between 10and 50 mm above the patient, between 50 mm and 75 mm, between 50 mm and100 mm, between 50 mm and 120 mm, between 50 mm and 130 mm, between 50mm and 140 mm, between 50 mm and 150 mm, between 100 mm and 200 mm,etc.). Accordingly, the field of view for various embodiments ofproximal cameras 520 on frames 510 can be different than the field ofview of distal cameras within a surgical site (e.g., distal cameras on asurgical tool). For example, for a distal camera within a surgical site,in order to provide an image from within a surgical site, a wide aspossible field of view is desired even if not in the center of field.Whereas for certain embodiments having a proximal camera 520 on a frame510, the field of view can be narrower because the camera 520 may not bein the surgical site 501.

Accordingly, certain embodiments of a proximal camera 520 on a frame 510can have a different optical function than a distal camera (e.g., adistal camera on a surgical tool). Some such embodiments of a proximalcamera 520 can be relatively small like an endoscope but behavestructurally like certain embodiments of a gimbal camera as describedherein for a surgical microscope view camera. In certain embodiments,the proximal camera 520 can be used with a camera providing a surgicalmicroscope view, such as for example described elsewhere in thisapplication. Accordingly, a distal camera can be useful for endoscopicprocedures. However, certain embodiments having a camera (e.g., aproximal camera and/or a more distal camera than a proximal camera)mounted to a frame 510 can be useful for procedures that are endoscopic,as well as those that are not critically endoscopic, but that aremicroscopic-like, such as for some brain surgical procedures.

In some instances, cameras positioned within the surgical site maycompromise the cameras' imaging ability, e.g., being in the wrongposition. For proximal cameras 520 mounted to a frame 510 being justoutside and adjacent to the surgical site 501, certain embodiments caninclude precision controls on the proximal cameras 520 that may not beplaced on small distal cameras inside the surgical site 501. For variousembodiments including a proximal camera 520, the frame 510 and mount 525(or clamp or finger) may remain still with only the gimbal 530 beingadjusted.

FIG. 14B1-a shows an illustration of an imaging system 540 comprising acamera 541, fiber optics 542 and a laparoscope 545 (which can also berepresentative of an endoscope) going inside the abdomen 546 through theabdominal wall 547 at a port 548 of entry (e.g., a trocar or cannulainsertion point). The laparoscope 545 can have a field of view 549 ofthe area 550 of interest. Because the laparoscope 545 goes inside thebody some distance, the port 548 of entry can become a rotation pointaround that fulcrum (e.g., at the insertion point). Whether thelaparoscope 545 is going through a single port, a laparoscope 545 andtools going through the same port, or the laparoscope 545 and toolsgoing through different ports, a lever arm on the laparoscope 545 and/ortools from the point 548 of entry in the imaging system 540 can createdisadvantageous imaging issues. Thus, having a lever arm from the port548 of entry can be an optical disadvantage for endoscopes andlaparoscopes. Further, a laparoscope 545 used through the abdominal wall547 is typically positioned at zero degree, with spin not desired. Forexample, rotation of the horizon with the laparoscope 545 is typicallynot desired. Because of the entry point 548 of the abdominal wall 547, agimbal system and/or imaging movement may generally be not possible incertain embodiments.

FIG. 14B1-b shows an illustration of certain embodiments of a medicalapparatus having one or more proximal cameras D on a frame G. Some suchembodiments can be useful where the surgical site opening is bigger andnot similar to an opening through an abdominal wall 547. Certain suchembodiments can have an optical advantage of no lever arm from an entrypoint of the body. Other optical advantages of some such embodimentsinclude the ability to use gimbals F similar to one's own vision (e.g.,eye, head, neck combination), a stationary ergonomic display, gimbalsfor stereo cameras, and/or planar four-bar mechanisms for positioningwithout disturbing horizontal positions of right and left eyeacquisition for horizontal viewing in display.

When not going through the abdominal wall 547, skull-based surgery,sinus surgery, knee surgery, or surgery in various other constrainedbody passages can be performed without a trocar opening. Often, suchsurgery (e.g., sinus surgery, neurosurgery, laparosurgery, or orthopedicsurgery) can involve rotation and result in disorientation. With suchrotation, the surgeon may have to keep the eye/brain combination workingto know one's location in space. Thus, in various instances, maintainingthe horizon without roll is desired.

In FIG. 14B1-a, when viewing the area 550 of interest with a laparoscope545, a surgeon may be restricted by the port 548 of entry through theabdomen 546. In FIG. 14B1-b, when viewing the area E of interest, amedical apparatus 600 including one or more proximal cameras D on aframe G, a gimbal system F can allow maintenance of the horizontal righteye/left eye configuration. A surgeon C can view through an ocular B toview an inside view of the body on the display A. The gimbal system Fcan comprise a mechanism configured to provide movement of the proximalcameras D while maintaining the horizon (e.g., right eye/left eyeparallel with the right eye/left eye in the display).

FIG. 14B2-a schematically illustrates imaging optics of an exampleimaging system compatible with certain embodiments of cameras asdescribed herein. FIG. 14B2-b shows an illustration of an exampletop-down view of certain embodiments disclosed herein. In certain suchembodiments, the center I of rotation is at the end of the gimbal of theproximal camera D instead of at the abdomen wall 547. In certainembodiments, such movement H can be analogous to rotating one's eyes intheir socket or rotating one's head around the axis of the neck.Movement laterally and longitudinally of the gimbal mechanism can occurat the center I of rotation. The rotation H around the center I (e.g.,side-to-side movement) can maintain the horizon in some embodiments.FIG. 14B2-c shows an illustration of an example side view of one opticalchannel of the apparatus shown in FIG. 14B2-b. In the example side view,the proximal camera D can rotate around J. Such movement can beanalogous to the head-and-neck combination (e.g., up-and-down movement).

FIG. 14B2-d shows an illustration of an example proximal cameraarrangement. Two proximal cameras D are shown here, e.g., at 12 o'clockand 6 o'clock on a plane of the frame G. There could be four cameras,e.g., also in an orientation of 3 o'clock and 6 o'clock and stillmaintain their stereo movement. FIG. 14B2-e shows an illustration of adisplay A viewable through portals (e.g., oculars B) by a surgeon C.FIG. 14B2-f illustrates a top-view of a left proximal camera DLproviding a left line LL of sight and a right proximal camera DRproviding a right line LR of sight. FIG. 14B2-f also illustrates anexample planar four-bar mechanism K that can allow movement of proximalcameras DL, DR in one orientation and then tip and turn without inducingroll. For example, yaw (movement around I), pitch (movement around J),and the motion (from the four-bar mechanism K) that can represent thelateral movement around in space are shown. In certain embodiments,motion for the proximal cameras DL, DR can include pitch and yaw, butnot roll. If there were roll on the acquisition cameras, theline-of-sight of the right eye and left eye in the display can bedifferent from the right eye and left eye of the acquisition system.FIG. 14B2-g shows an illustration of the side view of FIG. 14B2-f. FIG.14B2-g shows pivots P of the four-bar mechanism K and mount M to theframe.

Some embodiments can include sensors and cameras that can beautomatically rotated if the right eye and left eye of the displaysystem is rotated (e.g., roll). In some such embodiments, if a cameratipped, the display can follow the camera up to a certain point. Forexample, if a camera tipped at 15 degrees, the display can tip 15degrees. If roll were induced in the acquisition system in the proximalcameras and the display were to follow the roll, the cameras providing asurgical microscope view underneath the display can stay constant incertain embodiments (or might roll as well). In some embodiments, thesurgical microscope cameras can roll with roll of the display (as maypotentially the proximal cameras). However, in many embodiments, theability for the proximal cameras (and/or the surgical microscope viewcameras) to roll may not be provided or may be limited to reducedisorientation.

FIG. 14B3 is an illustration of certain embodiments described hereinshowing an oblique camera orientation. The motion F can allow one ormore proximal cameras D to image from multiple views. In someembodiments, unlike the endoscope or laparoscope 545 where pitch can betypically dictated by the port 548 of entry, the pitch is about thepoint J of rotation at a proximal camera D. Such motion F can be similarto an eye locating in a socket. Motion F can represent the ability of aproximal camera D to move around its own axis, supported by the frame G(e.g., on a non-laparoscopic device). The one or more proximal cameras Dcan include a relatively small pair of cameras such as stereo cameras.Providing only yaw, only pitch, a four-bar mechanism, and a z motion canmaintain the horizontal line through the lines of sight of the left andright cameras, eliminating roll in some embodiments.

As described herein, certain embodiments can include the x and y motionof a four-bar mechanism. In other embodiments, other x-y-z mechanismsbesides a four-bar mechanism can be used. In some embodiments, the pitchand yaw can be moved to a proximal camera on a frame independent of adisplay. By providing one or more proximal cameras with gimballingmotion, and in some embodiments, with all motions except roll, near theopening of the body, various embodiments can be different thanendoscopes and not comparable with microscopes.

An endoscope typically can have a 50 degree field of view to 110 degreefield of view. An operating room microscope (e.g., a camera providing asurgical microscope view) can have typically a smaller field of view,for example, the area of interest can be 50 mm in diameter and 300 mmaway from the vertex of the acquisition system.

In some embodiments, a proximal camera as described herein can be inbetween those two ranges. Thus, some embodiments of a proximal cameracan be described as being sometimes endoscope-like and sometimesmicroscope-like. Some such proximal cameras can have a narrower field ofview than an endoscope and a wider field of view than a microscope. Somesuch proximal cameras can also be positioned closer to the patient thana surgical microscope because the sensors and cameras can be muchsmaller. In addition, compared to loupes, various embodiments ofproximal camera and display can be more advantageous by having theopportunity to be in an ergonomic position and the ability to seeadditional views.

An endoscope, sinus scope, neuro scope, and/or laparoscope might have aworking distance from 10 mm to 100 mm, and might have a field of viewbetween 50 degrees and 110 degrees. An operating room microscope mighthave a field of view as seen on the patient (e.g., not in an angledspace) from 35 mm to 200 mm, and at a working distance of 200 mm to 450mm. In various embodiments, proximal cameras can be from 10 degree fieldof view to 50 degree field of view with working distances of 5 mm to 50mm, 40 mm to 100 mm, 40 mm to 150 mm, 50 mm to 100 mm, 50 mm to 150 mm,100 mm to 200 mm (e.g., something that allows you to be close to thepatient). In some embodiments, the workspace can be determined by theopening into the body, the depth of the wound or the surgicalpassageway, and the surgeon's hands and tools. Surgeons typically maynot take a tool like forceps with scissor action in their hand, but maytake a tool like pistol grips. In addition, surgeons typically may notrun their hands into the body. Accordingly, there may be a standoffworking distance, e.g., a 150 mm tool, at a minimum, for a 100 mmpassage. In addition, there may be some surplus of space between thesurgeon's hand and the opening of the body, e.g., to accommodate a drapefor example. The frame (e.g., 125 mm in diameter) can be positionedabove this working distance, e.g., within 10 mm, 25 mm, 40 mm, 50 mm, 75mm, 100 mm, 130 mm, 140 mm, 150 mm, 175 mm, or 200 mm in some instances,above the surgical opening to have this proximal camera. Thus, certainembodiments of proximal cameras can be disposed at the opening of thepassageway and view down in the surgical opening, with a narrower fieldof view than an endoscope, but a closer working distance than amicroscope. In some embodiments, the working distance can be between 1mm to 50 mm, 5 mm to 40 mm, 5 mm to 50 mm, 10 mm to 25 mm, 10 mm to 40mm, 10 mm to 50 mm, 50 mm to 75 mm, 50 mm to 100 mm, 50 mm to 120 mm, 50mm to 130 mm, 50 mm to 140 mm, 50 mm to 150 mm, 100 mm to 200 mm, 100 mmto 300 mm (or any ranges formed by any values between 1 mm and 300 mm).In addition, in some embodiments, the field in the object plane can bebetween 25 mm to 200 mm or between 25 mm to 250 mm (or any ranges formedby any values between 25 mm and 250 mm).

Additionally, any of the features or embodiments described in connectionwith the surgical tools, surgical visualization systems and componentsthereof, may be used with, combined with, incorporated into, beapplicable to, and/or are otherwise compatible with one or moreembodiments of a medical apparatus including one or more proximalcameras mounted to a frame disposed above the patient and/or surgicalsite as described herein.

For example, in various embodiments of a medical apparatus can includeone or more cameras. At least one of the cameras can include a surgicalmicroscope camera configured to provide a surgical microscope view ofthe surgical site. In various embodiments, the surgical microscopecamera is not coupled to a direct view surgical microscope. As alsodescribed herein, the medical apparatus can include a binocular viewingassembly comprising a housing and a plurality of portals (e.g. oculars).The plurality of portals or oculars (e.g., separated left and rightportals or oculars) can be configured to provide views of at least onedisplay disposed in the housing. The left and right portals (e.g.oculars) can be separated by sidewalls, baffling, tubing, etc. thatreduces optical cross-talk therebetween. For example, light from adisplay or display portion associated with the left portal or ocular isblocked so as to not propagate into the right portal or ocular, and viceversa. Similarly, light from a display or display portion associatedwith the right portal or ocular is blocked so as to not propagate intothe left portal or ocular. The medical apparatus can further include animage processing system (e.g., comprising processing electronics) incommunication with the camera (e.g., the camera including the surgicalmicroscope camera) and the one or more displays. As also describedherein, the image processing system can be configured to receive imagesacquired by the camera (e.g., the camera including the surgicalmicroscope camera), and to present output images based on the receivedimages on the one or more displays so that the output images areviewable through the plurality of oculars.

Mobile Display Devices

As described herein, various embodiments of a medical apparatus canswitch between and/or combine (e.g., dispose as adjacent to one another,tile, overlap, superimpose, dispose as PIP, etc.) images from differentsources. For example, the images can include a view of the surgical site(e.g., a surgical microscope view, an image from an endoscope, an imagefrom a proximal camera, an image from a surgical tool, an image from acamera on a retractor, etc.) combined with other information such as adata file, a computed tomography (CT) scan, a computer aided tomography(CAT) scan, magnetic resonance imaging (MRI), an x-ray, ultrasoundimaging, fluorescence imaging, neuro-monitoring, vital sign monitoring,etc. Such information can be images taken in real time or can be storedon a device such as a general all-purpose computer. The device can be adesktop computer, a laptop, or a notebook, etc.

The device can include equipment belonging to the hospital. However,some surgeons (or other medical personnel) may travel between differenthospitals and/or offices and have all patient records/data on one's owndevice. Accordingly, in certain embodiments disclosed herein, the devicecan include a surgeon's own portable equipment. For example, in certainembodiments as disclosed herein, the medical apparatus can includeinformation from a mobile display device such as a cellular telephone(e.g., a “cell phone” such as a smartphone), a tablet, an Internetenabled portable device, a network connected portable device, or anymobile device with a display capable of displaying images including text(e.g., devices known in the art or yet to be developed). The mobiledisplay device can include software applications (e.g., for navigationalguidance and ergonomic control), pre-recorded information, and memoryfor recording and storing information from other devices. The mobiledisplay device can include 3D volume data sets (e.g., functional MRI,CT, etc.), as well as 2D information.

In certain embodiments, it may be beneficial for a surgeon (or othermedical personnel) to view additional information during surgery withouthaving to take one's eyes away from the surgical site (e.g., from theviewing assembly described herein). In one example, the surgeon can viewadditional information about the same patient undergoing surgery. Asanother example, in situations where the surgeon could be in theoperating room for very long hours, the surgeon can view additionalinformation concerning another patient and/or other important affairs.The additional information can include an e-mail message, a textmessage, a medical communication, medical data, news, financial data,business information, business data, photographs, etc. For example, thesurgeon's mobile display device may include pre-recorded clinical imagesfrom medical devices, reference data, navigational device data in realtime, streaming text of patient data, hospital or other medicalcommunication, Twitter feeds, and/or other feeds through wired orwireless communication. In some embodiments, the mobile display devicecould include pico projectors as described herein for projectingfiducials or virtual touch screen images (e.g., buttons, icons,thumbnails, etc.) for the surgeon to see. As described above, someembodiments include a virtual touch screen where the surgeon's or user'smovement in free space is detected. The surgeon could move his/her handor instrument so as to touch an image, e.g., a virtual button, icon,thumbnail, etc. The surgeon's movement could be tracked to recordselection of an option and/or to provide an input via the virtual touchscreen.

Accordingly, various embodiments of a medical apparatus are configuredto allow the surgeon or medical personnel to view an image of thesurgical site and additional information from a mobile display devicesimultaneously. In addition, since the mobile display device has aseparate controller (e.g., processing electronics), certain embodimentsdescribed herein may reduce latency. In some such embodiments, themedical apparatus can include a docking station configured to receivethe mobile display device.

FIG. 15 shows an example embodiment of such an apparatus. The medicalapparatus 4100 can include a first display 4211 a (or display portion)configured to display a first image of a surgical site. The medicalapparatus 4100 can also include a controller (e.g., processingelectronics) configured to receive one or more signals corresponding tothe first image from a camera (not shown) and to drive the first display4211 a to produce the first image. Each signal path from the camera tothe first display 4211 a can include a control unit to control variousparameters (e.g., brightness, intensity, gamma, chroma, area ofinterest, subtraction, edge enhancement, text forms, graphics, etc. orcombinations thereof). The camera can include a camera as describedherein. For example, in some embodiments, the camera can include acamera providing a surgical microscope view. The camera can have a workdistance between 150 to 400 mm or to 450 mm. Other values outside theseranges are also possible. The camera can provide an image of a field (orlateral dimension). In some other embodiments, the camera can be avisualization device such as an endoscope, a proximal camera, a cameradisposed on a surgical tool, a camera disposed on a retractor, etc.

The medical apparatus 4100 can also include a docking station 4230(shown in FIG. 18) configured to receive a mobile display device (e.g.,a cell phone or tablet). The docking station can be disposed on theviewing assembly or at a distance away from the viewing assembly. Thelocation/orientation of the docking station 4230, e.g., with respect tothe viewing assembly, is not limited. The docking station 4230 can be inelectrical and/or optical communication with the mobile display device.For example, the docking station 4230 can include a port for electricaland/or optical communication with the mobile display device. The dockingstation 4230 can also include a port providing power to the mobiledevice. In some embodiments, the docking station can be in wirelesscommunication with the mobile display device. As shown in FIG. 15, themobile display device can include a second display 4212 a (or displayportion) having a second image. The second image can include informationas described above (e.g., e-mail message, a text message, a medicalcommunication, medical data, news, financial data, business information,business data, photographs, etc.) from the mobile display device.

In some embodiments, the surgeon sees the second display 4212 a througha set of lenses and/or mirrors or other optical elements so that thesurgeon can be at or near the patient, while the location of the mobiledisplay device and docking station is not particularly limited. Similarto the embodiments shown in FIGS. 12A-13C, a beam combiner 4230 a can beconfigured to receive and combine the first image of the surgical siteor at least a portion thereof and at least a portion of the second imagefrom the mobile display device. Various embodiments of the medicalapparatus 4100 can include imaging lenses for imaging the displays. Forexample, as shown in FIG. 15, one or more imaging lenses 4235 can bedisposed between respective beam combiners 4230 a and displays 4211 a,4212 a. In various embodiments, pellicle mirrors can also be used.

In certain embodiments, the combined images can be viewed within ahousing of a viewing assembly through a first ocular 4241 a. In someexamples, as shown in FIG. 15, the combined images can be viewed throughthe first ocular 4241 a to provide a left-eye view. In other examples,the combined images can be viewed through the first ocular 4241 a toprovide a right-eye view. The combined images can be viewed as apicture-in-picture (PIP) as shown in FIG. 13A or as adjacent to oneanother as shown in FIG. 13C, or can facilitate easy switching from oneimage to another. For example, in some embodiments, the medicalapparatus may darken an image while allowing another image to bevisible. Shutters can also be used in some embodiments to block orattenuate one image and not block or not attenuate (or attenuate less)the other image.

In various embodiments, the second image from the mobile display devicecan change between different images. As an example, the second image caninclude medical data shown adjacent to the first image of the surgicalsite or as a PIP or in the same field of view but spaced apart e.g., byborders. During surgery, the surgeon could receive notification of anemergency text message. In some such embodiments, since the latency ofthe second image from the mobile display device may be less criticalthan the first image of the surgical site, the surgeon or other medicalpersonnel could switch the second image from the medical data to thetext message. For example, some embodiments can be configured to allowthe second image to automatically switch to the text message or to aportion of the text message (e.g., while still displaying the firstimage). As another example, some embodiments can be configured to allowthe text message or a portion of the text message to be viewed with thesecond image as a PIP. In some such embodiments, if desired, the surgeonor other medical personnel could switch the second image from themedical data to the text message (e.g., while still displaying the firstimage).

Since the surgery room is a sterile environment, in certain embodiments,the medical apparatus 4100 can include a remote control configured tocontrol the mobile display device. As an example, the remote control canbe configured to control the second image displayed on the seconddisplay 4212 a of the mobile display device. For example, the remotecontrol can allow the user to switch the second image and/or adjust thesize, contrast, brightness, zoom, etc. The remote control can be anyuser interface which allows control of the mobile display device, forexample, from a distance away from the device (e.g., wirelessly and/orwithout the user physically touching the device). In some embodiments,the remote control can be disposed on or near the viewing assembly andcontrols thereof used by the user. For example, the remote control caninclude one or more handgrips on either side of the viewing assembly.The one or more handgrips can include buttons, a joystick, etc.Alternatively, the remote control can include a joystick (e.g., withbuttons), a handle, buttons, haptics, a touchpad, etc. In someembodiments, the remote control or another control can control the firstimage. Such control can control the camera providing the first image orcontrol the first display producing the first image. For example, thecontrol can adjust size, contrast, brightness, zoom, iris size, autogain, illumination, etc., e.g., using a series of buttons on a handlesuch as a handle on the binocular display assembly.

With continued reference to FIG. 15, the medical apparatus 4100 canfurther include a third display 4211 b (or display portion). The thirddisplay 4211 b can be configured to display a third image. In someembodiments, the third image can be viewed within the housing of theviewing assembly through the second ocular 4241 b. The medical apparatus4100 can include another controller (e.g., additional electronics asdescribed herein) configured to receive one or more signalscorresponding to the third image from a camera and to drive the thirddisplay to produce the third image. As an example, the third image cancomprise another image of the surgical site, e.g., from a cameraproviding a surgical microscope view, from a visualization device suchas an endoscope, from a proximal camera, from a camera disposed on asurgical tool, from a camera disposed on a retractor, etc. The first andthird displays 4211 a, 4211 b can be configured to provide 3D viewing ofthe images of the surgical site through the first and second oculars4241 a, 4241 b. For example, the cameras can provide different viewsfrom different stereo perspectives to produce 3D visualization for aviewer viewing the first and third displays 4211 a, 4211 b through thefirst and second oculars 4241 a, 4241 b. In some embodiments, theviewing assembly can be ergonomically decoupled from the camera thatprovides video to the first and/or third display 4211 a, 4211 b. Asdescribed herein, in various embodiments, the viewing assembly of themedical apparatus 4100 does not provide a view of the surgical sitethrough the first and second oculars 4241 a, 4241 b via an opticalpathway that passes through the housing. For example, certainembodiments are not associated with a direct view surgical microscope.

In certain embodiments, the mobile display device can operate as acontroller, processor, or computer, for the surgical visualizationapparatus, possibly assisting for example, in controlling variouscomponents (e.g., cameras, motors for orienting cameras, displays,lighting, etc.), performing processing (e.g., imaging processing,graphic user interface control, etc.) and or other electronic functionsincluding electronic computing or process functions. In someembodiments, the docking station 4230 can be configured to allow themobile display device to provide navigational guidance and/or ergonomiccontrol. For example, using a software application residing on themobile display device, a user can interact with the mobile displaydevice communicating through the docking station (e.g., via a port) orwirelessly to the medical device to guide the camera units, controltheir orientation, selection of cameras or view, or other. In addition,the docking station 4230 can be configured to allow the portion of thesecond image from the mobile display device to be displayed on anexternal monitor. Furthermore, in various embodiments, the dockingstation 4230 can be configured to allow the mobile display device toreceive and record images, e.g., of the surgical site, using the mobiledisplay device's camera. The mobile display device can also receive andrecord images from the other displays. In some embodiments, suchfunctions can be controlled by a remote control (e.g., using controls onthe viewing assembly) as described herein so as to not need to movelocations and/or to touch the mobile display device during surgery.

FIG. 16 shows another example embodiment of a medical apparatus 4100.While the schematic of FIG. 15 illustrates that certain embodiments ofthe medical apparatus 4100 can provide a mono view (as opposed to astereo view) of the image from the mobile display device in one eye(e.g., in the first ocular 4241 a), certain embodiments can provide amono view of the image from the mobile device in both eyes. As shown inFIG. 16, the image from the mobile display device can be combined withboth the first image of the surgical site from the first display 4211 aand the third image of the surgical site from the third display 4211 band be viewable within the housing of the viewing assembly through thefirst ocular 4241 a and second ocular 4241 b respectively. For example,the medical apparatus 4100 can include a beamsplitter 4240 configured toseparate and direct at least a portion of the second image from themobile display device to two optical paths (e.g., toward the first andsecond oculars 4241 a, 4241 b). The medical apparatus 4100 can include abeam combiner 4230 a configured to receive and combine at least aportion of the second image from the mobile display device and at leasta portion of the first image of the surgical site for viewing throughthe first ocular 4241 a. The medical apparatus 4100 can also include oneor more additional beam combiners 4230 b configured to receive andcombine at least a portion of the second image from the mobile displaydevice and at least a portion of the third image of the surgical sitefor viewing through the second ocular 4241 b. The combined images can beswitched in and out of the field of view, viewed as adjacent to oneanother in the field of view (e.g., space apart but within the field ofview), or viewed as a PIP.

FIGS. 17 and 17B show another example embodiment of a medical apparatus4100. In this example, the medical apparatus 4100 can provide a stereoview of an image from the mobile display device in both eyes (e.g.,using two oculars). Such embodiments can include an imaging lens 4235configured to provide two different views of the image from the mobiledisplay device, e.g., from two different angular perspectives. In someembodiments, two imaging lenses can be used to provide images from themobile display device, as shown in FIG. 17B. The medical apparatus 4100can include a beam combiner 4230 a configured to receive and combine animage of the surgical site from a first display 4211 a with at least aportion of the image from the mobile display device (e.g., one view fromthe second display 4212 a of the mobile display device). The medicalapparatus 4100 can also include another beam combiner 4230 b configuredto receive and combine a third image of the surgical site from a thirddisplay 4211 b and at least a portion of the image from the mobiledisplay device (e.g., a different view from the second display 4212 a ofthe mobile display device). The images from the mobile display devicecombined with the images from the first and third displays 4211 a, 4211b can be different perspectives of the image from the mobile displaydevice. The different perspectives can be different stereo views of theimage from the mobile display device such as to provide 3D viewing. Thecombined images can be switched in and out of the field of view, viewedas adjacent to one another in the field of view (e.g., space apart butwithin the field of view), or viewed as a PIP.

FIG. 17 shows an example embodiment utilizing a single imaging lens 4235to provide two different angular views of an image from the mobiledisplay device. Some embodiments can include multiple imaging lenses4235, e.g., an imaging lens for the left eye view and another imaginglens for the right eye view. Furthermore, in some embodiments, thedisplay 4212 a of the mobile display device can provide multiple images,e.g., a split screen providing the left and right eye views.

As shown in FIGS. 15-17B, the second display 4212 a is a portion of thedisplay of the mobile display device. In some such embodiments, themedical apparatus 4100 can include an optical pathway between the seconddisplay 4212 a and beam combiner 4230 a or 4230 b such that the beamcombiner 4230 a or 4230 b is capable of optically receiving the portionof the second image from the second display 4212 a. In otherembodiments, as shown in FIG. 18, the medical apparatus 4100 can includea fourth display 4212 b to display the second image or a portion of thesecond image. For example, the medical apparatus 4100 can include acontroller (e.g., electronics) configured to receive one or more signalscorresponding to the second image or a portion of the second image anddrive the fourth display 4212 b to produce the portion of the secondimage. In some such embodiments, the docking station 4230 can transmitthe signals from the mobile display device to the fourth display 4212 b.A beam combiner 4230 a can be configured to receive the second image ora portion of the second image from the fourth display 4212 b. Forexample, as shown in FIG. 18, the beam combiner 4230 a can be configuredto receive and combine the first image of the surgical site from thefirst display 4211 a and the portion of the second image from the fourthdisplay 4212 b.

In addition, similar to FIG. 15, a first image from the first display4211 a and a third image from a third display 4211 b can comprise imagesof the surgical site, e.g., from a camera providing a surgicalmicroscope view, from a visualization device such as an endoscope, froma proximal camera, from a camera disposed on a surgical tool, from acamera disposed on a retractor, etc. The first and third displays 4211a, 4211 b can be configured to provide 3D viewing of the images of thesurgical site through the first and second oculars 4241 a, 4241 b. Forexample, the cameras can provide different views from different stereoperspectives to produce 3D visualization for a viewer viewing the firstand third displays 4211 a, 4211 b through the first and second oculars4241 a, 4241 b.

Certain embodiments can include additional beamsplitters, beamcombiners, mirrors (e.g., pellicle mirrors), lenses, and/or displays.For example, as shown in FIGS. 20-22, the medical apparatus 4100 caninclude a fifth display 4211 c configured to display a fifth image. Themedical apparatus 4100 can include an additional controller (e.g.,electronics as described herein) configured to receive one or moresignals corresponding to the fifth image from a camera and to drive thefifth display 4211 c to produce the fifth image. In some instances, thefifth image can include another image of the surgical site. The medicalapparatus 4100 can include another beam combiner 4230 c configured toreceive and combine the first, second, and fifth images or portions ofthe first, second, and fifth images for viewing within the housing ofthe viewing assembly through the first ocular 4241 a. The medicalapparatus 4100 can also include a sixth display 4211 d configured todisplay a sixth image, a controller (e.g., electronics as describedherein) configured to receive one or more signals corresponding to thesixth image from a camera and to drive the sixth display 4211 d toproduce the sixth image, and/or a beam combiner 4230 d configured toreceive and combine at least the third and sixth images or portions ofthe third and sixth images for viewing within the housing of the viewingassembly through the second ocular 4241 b. In some instances, the fifthand sixth displays are configured to provide stereo images to provide 3Dviewing of images of the surgical site through the first and secondoculars 4241 a, 4241 b. The cameras can provide different views fromdifferent perspectives of the surgical site to produce 3D visualizationfor the viewer viewing the fifth and sixth displays through the firstand second oculars 4241 a, 4241 b. In other embodiments, the images fromthe additional display portions can include other information such as adata file, a computed tomography (CT) scan, a computer aided tomography(CAT) scan, magnetic resonance imaging (MRI), an x-ray, ultrasoundimaging, fluorescence imaging, information from a mobile display device,neuro-monitoring, vital sign monitoring, etc. The images from thedifferent displays can be switched between different images, disposedadjacent to one another, or disposed as a PIP.

Various embodiments described herein utilizing images from mobiledisplay devices have the advantage of having a separate controller forthe image displayed by the mobile display device, which may reducelatency. Certain embodiments described herein can be applied to aprimary display, a surgeon display, an assistant display, possibly otherdisplays, or any combination of these. The images that the surgeon seescan also be projected onto other displays for other medical personneland staff to see.

FIG. 23 shows an example embodiment of a mobile display device incommunication with a docking port disposed on a binocular display unit.The docking port may comprise an electrical connection for electricalcommunication with other components in the system. Thelocation/orientation of the docking port and/or mobile display devicewith respect to the binocular display unit is not limited. In addition,additional docking ports can be disposed on the binocular display unitor at a distance away from the binocular display unit. For example, invarious embodiments, a docking port can be provided for the mobiledisplay device of an assistant surgeon (or other medical personnel). Insuch embodiments, images from the assistant surgeon's mobile displaydevice can be displayed on the assistant display. In some embodiments,the assistant surgeon can view images combined with other images or canswitch between images without changing the images seen by the primarysurgeon.

Furthermore, FIG. 24 shows an example headset or head mounted displaythat can be used to view images from a mobile display device (e.g., anOculus Virtual Reality type display), alone or in combination with otherimages from other displays or devices. The headset or head mounteddisplay may comprise an immersive display and/or other near eye display.In some such embodiments, the display of the mobile display device canprovide multiple images, e.g., a split screen providing left and righteye views, for viewing through the left and right eyepieces of theheadset.

In various embodiments, the docking port can be included on any side ofthe binocular viewing assembly. For example, with reference to FIGS. 23and 24, the docking port can be on the side opposite to the side wherethe docking port is shown in FIGS. 23 and 24. The docking port can be onthe side opposite to the side where oculars are closest. Similarly,multiple docking ports can be included. For example, one docking portmay be on any 1, 2 or 3 of these sides. Such configuration accommodatesone or more assistants positioned opposite of the surgeon or on thesurgeon's left or right or any combination of these locations.

FIG. 25 shows another example headset or head mounted display that canbe used to view images from a mobile display device. This example issimilar to the embodiments described with respect to FIG. 24, yet thedocking port is on the same side of the binocular display unit (andslightly above) as the primary surgeon's oculars. Such embodiments canallow the primary surgeon to view the images on the mobile displaydevice without moving far from the surgical site. Various embodimentscan include a headset for the primary surgeon in combination with one ormore assistant headsets as shown in FIG. 24. As described herein,various elements and combinations of elements for viewing multipleimages from one or more displays, e.g., including one or more mobiledisplay devices, are possible.

Remote Control

As described herein, certain embodiments of a surgical visualizationsystem can include a control (e.g., disposed on the viewing assembly)configured to control a camera, a display, mobile display device, orother medical device. For example, as described herein, the control canadjust size, contrast, brightness, zoom, iris size, auto gain,illumination, etc. In some embodiments, the control can include a pairof handgrips with buttons on either side of the viewing assembly so thatthe surgeon would not need to move away from the surgical site. In someembodiments, the control can include a single handgrip with buttonswhich allows the surgeon to not have to move both hands from thesurgical site even momentarily.

In further embodiments, the control can allow hands free control so thatthe surgeon would not need to put down surgical tools from both hands.Such hands free controls can include a virtual touch screen as describedherein and/or voice command recognition. Such hands free controls canalso include eye and/or head tracking. For example, the control caninclude eye and/or head tracking systems that monitor the eye movementsand/or head movements of the user. An electronic processing system incommunication with the control can include algorithms that can interpretthe user's eye movements and/or head movements. The control can beconfigured to vary at least one parameter (e.g., size, contrast,brightness, zoom, iris size, auto gain, illumination, etc.) of a camera,a display, mobile display device, or other medical device based on theeye movements and/or head movements of the user to adjust the displayedimages. In some such embodiments, the control can be configured tomeasure the position and/or movement of one or more of the user's eyes,and/or measure the position and/or movement of the user's head. Forexample, in some embodiments, the control can monitor the user's gazeinteraction with the surgical site, a virtual screen with icons, and/orother object. Based on such interactions, e.g., number of gazes, periodof gaze, winking, etc., the control can control the camera, display, orother medical device that the user's gaze is directed towards.

Efficient Surgical Imaging Systems

Various surgical microscopes can provide a direct view of the surgicalfield. In such microscopes, the visible light from the illuminationsource is typically split (e.g., via a beamsplitter) between the directview and the camera. Accordingly, the amount of visible light providedto the camera is reduced, which can affect the quality of the images(e.g., brightness) as seen by the user.

As disclosed herein, certain embodiments of a surgical imaging systemproviding a surgical microscope view can be configured to be decoupledfrom the camera providing the video to the display, e.g., FIGS. 2 and3A. In various such embodiments, since the camera is decoupled from theviewing assembly, a large portion (e.g., substantially all) of the lightfrom a light source can be directed to the camera without the need for abeamsplitter, which can also result in light loss. As such, variousembodiments described herein can provide highly efficient surgicalimaging systems compared to direct-view surgical microscopes. Qualitywell lit images can be provided without needing excessive illuminationof the patient.

High Intensity Displavs

As described herein, certain embodiments can include a plurality ofdisplays (or display portions), e.g., a plurality of displays 2211 a,2212 a for a left-eye view and a plurality of displays 2211 b, 2212 bfor a right-eye view as shown in FIGS. 12A-12B. The displays can beilluminated by a source of illumination (e.g., a fiber optic, LED, etc.)or can be emissive displays. In some embodiments incorporating opticalelements (e.g., beamsplitter, beam combiners, etc.), the image viewedthrough an ocular can have a reduction in the screen brightness.Accordingly, in various embodiments, the displays are configured to bebrighter than certain conventional displays to compensate for loss inpassing through optical elements like beamsplitters and beam combiners.Certain embodiments of the medical apparatus described herein, forexample, include one or more displays having a relatively highbrightness. In certain embodiments, the medical apparatus can includeone or more light sources to illuminate a display. The one or more lightsources can provide a power between about 0.5 watt to about 10 watts(e.g., 0.5 watt, 1 watt, 2 watts, 3 watts, 4 watts, 5 watts, 6 watts, 7watts, 8 watts, 9 watts, 10 watts of power, or any value therebetween)or any ranges in between (e.g., 2-4 watts, 3-7 watts, 5-6 watts, etc.).Consequently, the output from one of the displays can be between about0.5 watt to about 10 watts (e.g., 0.5 watt, 1 watt, 2 watts, 3 watts, 4watts, 5 watts, 6 watts, 7 watts, 8 watts, 9 watts, 10 watts of power,or any value therebetween) or any ranges in between (e.g., 2-4 watts,3-7 watts, 5-6 watts, etc.). In some embodiments, the light source(s)(e.g., an LED) can provide the higher power instead of the typical power(e.g., about 250 mW). In some other embodiments, the one or more lightsources can include one or more auxiliary light sources (e.g., anadditional LED backlight, edge light, or frontlight) that provideadditional power. As one example, the display can include a backlit oredge lit display panel illuminated continuously by a plurality of LEDson the back side or edge of the panel with power such as 0.5 W, 1 W, 2-4W, or 5-6 W. Such light sources(s) can be run off a battery or linevoltage. Utilizing one or more light sources to provide more power canprovide certain embodiments with displays having sufficient and/orrelatively higher brightness.

Neuro-Monitoring and Vital Sign Monitoring

As described herein, various embodiments of a medical apparatus caninclude a view of the surgical site (e.g., from a camera providing asurgical microscope view, from a visualization device such as anendoscope, from a proximal camera, from a camera disposed on a surgicaltool, from a camera disposed on a retractor, etc.) combined with otherinformation such as a data file, a computed tomography (CT) scan, acomputer aided tomography (CAT) scan, magnetic resonance imaging (MRI),an x-ray, ultrasound imaging, fluorescence imaging, information from amobile display device, etc. As described herein, the medical apparatuscan include additional information stored in a device or received inreal time. Furthermore, the medical apparatus can include a view ofinformation providing neuro-monitoring, e.g., using neurologicalmetrics. For example, the information can include electroencephalography(EEG) to record electrical activity of the brain. As another example,the information can include electromyography (EMG) to record electricalactivity of muscles to evaluate the muscles and the nerves that controlthe muscles. The information can also include a monitor of vital signs,such as the heart rate, blood pressure, body temperature, weight, etc.In some embodiments, the information provided in various embodiments caninclude a graph of signals as a function of time. In some otherembodiments, the information can be colored light, an LED light, asignal, graphics, or text when a certain threshold has been reached asthe surgeon takes action during the procedure. With neuro-monitoringand/or vital sign monitoring in real time, the surgeon and/or othermedical personnel can receive direct feedback during the surgery, e.g.,when surgery is near a nerve, or otherwise to give biofeedback or statusof vital signs. The information can viewed on a primary display, surgeondisplay, and/or assistant display, etc. Neuro-monitoring information canbe provided on a separate display, a same display as the image or aseparate display and a beam combiner as described herein.

Pair of Mobile Display Devices

As described herein, various embodiments of a medical apparatus, e.g., abinocular display, can provide a stereoscopic view of the surgical siteusing a pair of displays (e.g., left and right displays) for viewingthrough a pair of oculars (e.g., left-eye view and right-eye). Someexamples are shown in FIGS. 9A-9B. Each of the pair of displays candisplay an image from a respective camera. One or more optical elementscan be included in each optical path to direct light from eachrespective display to the respective ocular. In certain embodimentsdescribed herein, a cell phone (or other mobile display device such as atablet) including a camera and display can be used for left and rightchannels. For example, the display of a first cell phone can be used asthe display for the left-eye view (e.g., to display the image for theleft-eye view). The display of a second cell phone can be used as thedisplay for the right-eye view (e.g., to display the image for theright-eye view). The cameras of the first and second cell phones canprovide the image of the surgical site to be displayed on eachrespective display (e.g., may be a surgical microscope having a workdistance between 150 and 450 mm). In some embodiments, one or moreoptical elements can be added to one or both of the cameras. Forexample, a lens could be added to provide increased or decreasedmagnification or work distance. In addition, one or more opticalelements (e.g., one or more mirrors, lenses, etc.) can be included ineach optical path to direct light from each respective cell phonedisplay to the respective ocular. The provided images can be differentperspectives of the surgical site to provide a stereo view (e.g., 3Dviewing) of the surgical site. In some embodiments as described herein,the cell phones can be controlled by a remote control (e.g., a handgrip,a joystick, a handle, buttons, haptics, a touchpad, etc.). In certainembodiments, also as described herein, each cell phone can have its owncontroller (e.g., processing electronics) to receive signalscorresponding to the image from its camera and to drive its display toproduce the image. Advantageously, having separate controllers for eachof the left and right cameras and displays may reduce latency.

Separate Display Controllers

As described herein, certain embodiments of surgical visualizationsystems may reduce latency by using one or more beam combiners, e.g.,FIGS. 12A-13C and 15-21, to receive images from displays and to combinethe images for viewing through oculars. In such embodiments, the imagesare provided optically for viewing. In various other embodiments, theimages can be provided electronically by separate controllers (e.g.,processing electronics) to separate displays in the left and right eyechannels. For example, different displays in the left and right eyechannels can have a separate dedicated controllers which can allow forreduced latency. Furthermore, with separate controllers for the separatedisplays, if one of the controllers were to fail, the controller in theother channel can continue to provide images to the surgeon. Likewisethe left and right eye channels have separate controllers which can alsoallow for reduced latency.

Multiple Displays in Field of View

As described herein, various embodiments of a medical apparatus canswitch and/or combine (e.g., dispose as adjacent to one another, tile,overlap, superimpose, dispose as PIP, etc.) images (including graphicsand/or text) from different sources as shown in FIGS. 13A and 13C. Insome such embodiments, the images can be presented within a centralportion of the view of an ocular. For example, the images can be visiblewithin a central rectangular portion within a circular or oval field ofview of an ocular because displays are typically rectangular. The shapesand sizes of the central portion and/or the field of view of the ocularare not particularly limited.

In some embodiments as shown in FIG. 22, the medical apparatus 2700 caninclude multiple displays 2701, 2702, 2703 within the field of view 2704of the ocular. In this example, a first image is displayed on a firstdisplay 2701. A second image is displayed on a second display 2702, anda third image is displayed on a third display 2703. A first controllercan receive one or more signals corresponding to the first image from acamera or display and drive the first display 2701 to produce the firstimage. A second controller can receive one or more signals correspondingto the second image from a camera or display and drive the seconddisplay 2702 to produce the second image. Furthermore, a thirdcontroller can receive one or more signals corresponding to the thirdimage from a camera or display and drive the third display 2703 toproduce the third image. Utilizing separate displays with separatecontrollers for each display may reduce latency. The example shown inFIG. 22 illustrates a first display 2701 as a central rectangulardisplay extending over a substantial portion of the view with the secondand third displays 2702, 2703 as smaller peripheral rectangulardisplays.

However, the displays can be in any location and have any shape, size,and/or aspect ratio. In addition, the displays can have the same ordifferent shape, size, and/or aspect ratio from each other. The numberof displays within the field of view is also not limited. For example,the number of displays can include 2, 3, 4, 5, 6, 7, 8, etc. Thedisplays can be, for example, an LED light, an LCD, an LED display, anOLED display, a DMD display, or an emissive display although otherdisplays are possible. The images can include a view of the surgicalsite (e.g., a surgical microscope view, an image from an endoscope, animage from a proximal camera, an image from a surgical tool, an imagefrom a camera on a retractor, an image from a mobile display device,etc.). The images can also include information such as a data file, acomputed tomography (CT) scan, a computer aided tomography (CAT) scan,magnetic resonance imaging (MRI), an x-ray, ultrasound imaging,fluorescence imaging, information from a mobile display device,neuro-monitoring, vital sign monitoring, graphics, text, etc.

As an example, an image of the surgical site can be presented on adisplay within a central portion of the field of view of the ocular.Another image can be presented on a smaller peripheral display outsidethe central portion but within the field of view of the ocular. Theperipheral image can be, for example, a text message provided by a cellphone. The cell phone can have for example, a four inch, five inch, sixinch, or any size display. The text message displayed on the cell phonecan be presented on a smaller (e.g., 1 inch, 1.5 inch, 2 inch, etc.)display within the field of view of the ocular. Accordingly, some suchembodiments can include a docking station as described herein,configured to receive the mobile display device. As also describedherein, since the latency of one of the images may not be as critical asthe image of the surgical site, certain embodiments can include aswitching system configured to switch an image to another image (e.g.,from another source). In some such embodiments, the displays can becontrolled by a remote control (e.g., a handgrip, a joystick, a handle,buttons, haptics, a touchpad, etc.). For example, the images can beswitch from one image from one source to another image from a differentsource. Thus, various embodiments include multiple displays in the fieldof view of a single ocular to provide multiple images simultaneouslywith reduced latency.

Certain embodiments can include another ocular having a plurality ofdisplays within the field of view. For example, the medical apparatuscan include first and second oculars (e.g., left-eye view and right-eyeview). As an example, both the left-eye view and right-eye view can havefour displays each within its field of view, for a total of eightdisplays. Various displays within certain embodiments of the medicalapparatus can be configured to provide 3D viewing of the images of thesurgical site through the first and second oculars. As described herein,in various embodiments, the medical apparatus can include a viewingassembly comprising a housing and the oculars. In some embodiments, themedical apparatus does not provide a view of the surgical site throughthe first and second oculars via an optical pathway that passes throughthe housing. For example, certain embodiments are not associated with adirect view surgical microscope. In some instances, the medicalapparatus can provide a mono view of the surgical site. Various examplesare possible.

Display Designs

Various embodiments of the binocular display assembly described hereinform a real image of first and second displays (e.g., LCD) displays.This real image is imaged onto the retina of a viewer (e.g., thesurgeon) with the aid of the oculars. FIG. 26 shows a system that doesnot form real image. Instead, a virtual image that is viewed by the eyeis formed by the optics. Similarly the system in FIG. 27 is notconfigured to form a real images, rather a virtual image that is viewedby the eye is formed by the optics.

In contrast, FIG. 28 illustrates an embodiment such as described hereinwherein a real image is formed by the imaging optics. This real image ofthe display is formed at the location of the field stop by the imagingoptics. In FIG. 28, this field stop is shown located between the imagingoptics and the oculars. A conjugate image of the real image located atthe field stop is formed onto the retina with the aid of the oculars.Advantageously, in various embodiments, the ocular can be adjusted fordoctors having impaired visual acuity such as myopia or hyperopia. Theoptical path distance through the ocular may be adjusted, for example,to bring the intermediate real image at the field stop in focus for thesurgeon even when the surgeon is not wearing corrective eyeglasses. Suchadjustments may be made without creating vignetting problems.

FIG. 29 illustrates another embodiment such as described herein whereinthe optics that images the display has an exit pupil where the image ofthe display is collimated. This portion of the system is referred to inthe drawing as an infinite conjugate section as collimated light isoutput by the optics. The exit pupil can match up with entrance pupil ofthe ocular or binocular assembly including the ocular. In this system, areal intermediate image of the display is formed at the field stop. Thisreal image is imaged onto the retina of the viewer with the aid of inthe binocular assembly. The binocular assembly including the oculars isreferred to as a binocular infinite conjugate section as the collimatedlight is received by this section. In some embodiments described herein,a mounting fixture enables the binocular assembly including the ocularto be connected to the display assembly that forms a collimated image ofthe display. As discussed above, in various embodiments, the ocular canbe adjusted for doctors having impaired visual acuity such as myopia orhyperopia. The optical path distance through the ocular may be adjusted,for example, to bring the intermediate real image at the field stop infocus for the surgeon even when the surgeon is not wearing correctiveeyeglasses. Such adjustments may be made without creating vignettingproblems.

In some embodiments, an eye box can be created that allows the eye tomove about a distance from the display assembly and still see the imageproduced by the display. FIG. 30 illustrates a configuration forproducing such an eye box. In this example, a cell phone is shown as thedisplay as is described herein, however, other displays may be employed.As shown in FIG. 30, a combination of optical elements forms acollimated beam, e.g., an image at infinity. The eye can focus this beamdown onto the retina and thereby see an image of the cell phone display.

As illustrated, multiple laterally disposed pupils are created such thatthe eye can see the display over a range of lateral positions. Excessivelateral movement may produce vignetting. However, the image will bevisible to the viewer for a range of head positons.

Three optical elements are shown in FIG. 30, each comprising reflectiveoptical elements (e.g., mirrors). However, a different number of opticalelements may be employed. Lens may also be used. A combination ofelements make up of one or more reflective elements and one or morelenses may also be used. The curvatures of the optical elements may beaspherics and may be freeform shapes.

In the configuration shown, a real image of the object (e.g., the cellphone) is formed between in the optical path between M1 and M3.

Such a display configuration may be useful for an assistant display.Such an assistant display can be provided, as described herein, forassistants and other allied health professional to view during surgery.The display may be located, for example, on the outside of the binocularviewing assembly that the primary surgeon views through or otherwise(e.g., on different arms and/or stands). The display can be in variouslocations/orientation including for example, 90°, 180°, −90° withrespect to the view of the surgeon viewing through oculars. One or moresuch display may be used. Such a display as shown in FIG. 30 can allowthe viewer to stand off at some distance and still see stereo. This maypermit the viewer to move more freely and provide increased situationalawareness.

In certain configurations, mirror M3 could be semitransparent, meaningyou could see the patient through the mirror. Such a design mightadditionally provide increased situational awareness.

In some embodiments, one or more of the optical elements could beadjustable, for example, to allow the user to adjust the distance atwhich the user can see the image.

Degrees of Freedom of Movement

In various embodiments described herein the binocular display assemblyis configured to move in five degrees of freedom as seen from theposition where the eyes meet the oculars. In particular, the binoculardisplay unit can translate up/down (x), left/right (y) and forward orbackward (z) and can pitch (tilt upward or downward) and yaw (rotateleft or right). In various embodiments, the binocular display unit andthe oculars cannot roll, that is rotate clockwise or counter-clockwiseas seen from the position where the eyes meet the oculars. Roll cancause the viewer, e.g., surgeon, to become nauseas. Accordingly, invarious embodiments the binocular display assembly (and oculars) arephysically restricted from roll movements. For example, arms from whichthe binocular display assembly is supported may restrict roll fromdegrees of freedom of movement and keep the combination of the left andright oculars horizontal (parallel with the floor).

Similarly in various embodiments described herein the stereo cameras(such as the cameras that provide surgical microscope views) areconfigured to move in five degrees of freedom as seen from the viewprovided by the stereo camera. In particular, the stereo cameras cantranslate up/down (x), left/right (y) and forward or backward (z) andcan pitch (tilt upward or downward) and yaw (rotate left or right). Invarious embodiments, the cameras cannot roll, that is rotate clockwiseor counter-clockwise as seen from the view provided by the cameras. Asdiscussed above, roll can cause the viewer, e.g., surgeon, to becomenauseas. Accordingly, in various embodiments the cameras are physicallyrestricted from roll movements. For example, the cameras may be disposedon arms, positioners and/or mounts that restrict roll from the degreesof freedom of movement and that keep the combination of the left andright camera views horizontal (parallel with the floor).

In various embodiments, the five degrees of freedom of movement of thebinocular display assembly and oculars are decoupled from the fivedegrees of freedom of movement of the surgical microscope view camerasand/or other cameras. Accordingly, in certain embodiments, the ocularsand/or binocular display assembly can move in five degrees of freedomwithout relying on movement of the surgical microscope view cameras orother cameras. Similarly, in certain embodiments, the surgicalmicroscope view cameras and/or other cameras can move in five degrees offreedom without relying on movement of the oculars and/or binoculardisplay assembly.

In various embodiments, mechanical fixtures can limited the degrees offreedom of movement of stereo cameras to, for example, avoid roll ofthat would cause the viewer to become nauseas. For example, mechanicalfixtures can limit the roll of stereo endoscopes. In some embodiments,for example, the stereo endoscope or other stereo camera can be affixedto a mount such as a ring in a manner to restrict roll and keep thehorizon of the stereo camera substantially constant. Other types ofguides that limit the stereo camera from rolling can also be used.Accordingly, for some embodiments, five degrees of freedom or less(e.g., one or more of x, y, z translation and pitch and yaw) may beprovided to the stereo cameras such as for stereo endoscope. Limiting tofive degrees of freedom instead of six degrees of freedom, such as byremoving roll and keeping the horizon substantially constant can reducediscomfort and disorientation for the viewer such as the surgeon orassistant.

Fixed Working Distance Objectives

In various embodiments, the camera configured to provide surgicalmicroscope views may include a microscope objective with a fixed workingdistance. This fixed working distance may correspond to a fixed focallength. This objective, may have for example a work distance or focallength, of 150 mm to 450 mm, such as 150 mm, 200 mm, 250 mm, 300 mm, 350mm, 400 mm, 450 mm, or values therebetween. Other work distances,including those described elsewhere herein may be possible. Although theobjective may have a fixed work distance, the objective may permitzoom/variable magnification. The camera may be configured to receivedifferent objectives having different working distances, which may beselected by the surgeon for different surgical procedures. One objectivewith a fixed working distance or focal length of 400 mm may be switchedout for another objective having a fixed working distance or focallength or 150 mm to accommodate different circumstances, different typesof surgery or based on the surgeons preference. Other objectives withdifferent working distances or focal lengths may be used. In variousembodiments, therefore the surgical microscope camera is configured tohave the objective conveniently be removed and replaced with anothermicroscope objective having a different working distance or focallength. The mechanical interface, for example, threading to screw theobjective in place, clamps, etc. may be configured to facilitate suchconvenient interchangeability. As described herein, in certainembodiments for a stereo system, light for both left and right channelspass through a single objective. In other embodiments, two objectives,one for the left channel and one for the right channel, may be used.

Laser Distance Positioning Guide

As discussed above, various embodiments include at least one camera thatprovides surgical microscope views. Some of these cameras have a workdistance between 175 mm and 450 mm. Certain embodiments are equippedwith a system for enabling the user to set the proper distance betweenthe cameras and the surgical site. The camera providing a surgicalmicroscope view may, for example, be suspended from an articulated armwhich enables the distance of the surgical microscope camera from thepatient and surgical site to be adjusted to match the working distanceof the surgical microscope camera. A system that informs the person(e.g., physician, nurse, technician, etc.) positioning the surgicalmicroscope camera with respect to the patient may be useful for givingthat person guidance as to whether to increase or decrease theseparation of the between patient and the surgical microscope viewcamera. Such a system may also aid in lateral alignment.

Some such systems rely on projecting beams of light onto the patient atthe surgical site. For example, lasers may be mounted on the assemblywith the left and right surgical microscope view stereo cameras. Thelasers may be configured to direct a pair of laser beams that convergeonto a spot at the proper distance between the surgical site and thecameras. Accordingly, if the distance between the camera(s) and thesurgical site is not correct, the beams will be incident on the patientat two spaced apart locations. Two separate laser beam spots will beformed on the patient. In contrast, if the distance between the patientat the surgical site and the camera that provide a surgical microscopeview are correct, the laser beams are configured to overlap. In someembodiments, one laser beam may be configured as a line oriented in onedirection while the other laser bean is configured as a line oriented ina perpendicular direction. Likewise, when the two laser beams overlap, acrosshair pattern is formed. In other embodiments, both beams may becross-hairs that overlap when the proper distance is established betweenthe patient and the surgical microscope view cameras. Otherconfigurations and beam patterns are possible. In some embodiments, thelaser outputs an infrared laser beam and is visible by viewing a displaysuch as in the binocular display assembly that is coupled to an infraredsensor that may form part of the surgical microscope view cameraassembly. In other cases, visible light sources can be used. The personpositioning the surgical microscope view camera with respect to thesurgical site may see the light beams on the patient without the aid ofa display

In some embodiments as shown in FIG. 31, light sources can emit lightthat passes through the objective and are thereby focused by theobjective down to a spot, which corresponds to the work distance and/orthe proper distance between the cameras and the surgical site. In somecases, the laser module output collimate beams that are parallel to eachother. These parallel beams are focused by the objective (and therebycaused to overlap) at a distance that corresponds to the focal lengthand, hence working distance, of the surgical microscope cameras.Accordingly, for such systems, if the objective is switched out, forexample, for another procedure, the beam will be focused by the newobjective at the focal length and working distance of that newobjective. This feature is convenient as the distance where the beamsconverge to a common location will chance to the proper location whenthe objective is switched out to a new objective for a differentprocedure or preference of the surgeon.

The systems described above that employ light beams such as laser beamsto establish the proper distance between the patient/surgical site andthe surgical microscope view cameras may also be configured to providefor laterally aligning the surgical microscope view camera as well. Thelocation where the beams overlap may, for example, correspond to thefield of view (e.g., the center of the field of view) of the surgicalmicroscope view camera. Thus, one can use the position where the beamsoverlap on patient to laterally align the surgical microscope camera aswell. Other laser beam pointing based systems can be use.

Other types of systems to aid in positioning the surgical microscopeview camera with respect to the patient may also be employed. Forexample, a range finder may be used to establish the proper distancebetween the patient and the surgical microscope view camera. A laserrangefinder may, for example, be incorporated into the assembly in someembodiments. In certain implementations, the output of the rangefindercan be displayed using the binocular display and be viewable by lookingthrough the oculars. In some implementations, the output of therangefinder may be viewed without looking through the oculars. Forexample, a display may be disposed on the outside of the binocularviewing assembly (e.g., on the housing) that provides distance orpositioning data based on the laser rangefinder. This display mayindicate whether to increase or decrease the separation and may showwhen the separation is correct and matched to the working distance ofthe surgical microscope view camera. A display located elsewhere mayalso be used. For example, this information may alternatively or inaddition be sent to a panel display. In certain embodiments, thisinformation may be projected onto a surface such as onto the patient forviewing.

Accordingly, in various embodiments, feedback for setting the properdistance (and possibly lateral alignment) may be provided by looking ata display in the binocular viewing assembly such as when looking throughone or both of the oculars. Additionally, feedback for setting theproper distance (and possibly lateral alignment) without looking throughone or both of the oculars and may be provided by looking at the patientor a display, for example, located on the binocular viewing assembly orelsewhere.

Software/Image Processing

In some embodiments as described herein, multiple displays can beutilized together to allow viewing of an image of the surgical field.The images can thereby be superimposed over each other. The multipledisplays and images can provide features that would be helpful for theuser or assist in the surgical procedures. For example, a two displaysystem can be used where one of the displays allows viewing of an imageof the surgical field. The second display can superimpose an image overthe first surgical field display using a beam combiner as described indetail herein. The ability to superimpose or display both images at thesame time utilizing the beam combiner as described herein can allow forthe elimination of the delay present when images are added through theuse of a computer processor. By allowing for the multiple displays to besuperimposed or adjacent to each other through combination of themultiple displays, the multiple displays can be viewed together as asingle real time image. This can allow for viewing of fiducials,drawings and/or annotations, virtual alignment of implants, and/ordisplay of other features in coordination with a display of an image ofthe surgical field. In various embodiments, the image of the fiducial,drawings, annotations, implants, etc., can also appear on one or moreassistant displays, and/or on a panel display viewable by others in theroom or by one or more displays located remotely. In some embodiments,an image can be projected directly onto the patient and/or surgicalsite. This can allow viewing of fiducials, drawings and/or annotations,virtual alignment of implants, and/or display of other features directlyonto the surgical site or patient's body. This method of projectingdirectly onto the surgical site may eliminate the need for the seconddisplay of the surgical site as described previously and utilize onlythe overlaid image and/or images onto the surgical site itself. In someembodiments, a projector system may including a small projector such asa handheld projector, a pico projector, mobile projector, pocketprojector, mini beamer, or other projector can be used.

In some embodiments, an image can be used to measure features of thesurgical field. The image can be superimposed or adjacent to a firstsurgical field display or projected adjacent to and/or on the surgicalfield itself. Fiducials as described herein can be used in the image tomeasure portions or positioning of multiple aspects of the surgicalfield. The fiducial markers can be a marking or pattern on the imagedisplayed, for example, the fiducial markers can be a mark, a line, aset of marks, a set of lines, image, and/or set of images. Other knownpoints or features can be used as a fiducial markers as well. In someembodiments, for example, the image can include a ruler or grid patternthat can be displayed adjacent to and/or superimposed on the firstdisplayed image of the surgical field or projected adjacent to and/or onthe surgical field.

The multiple images or a single image can be calibrated and/or scaled tothe same size or to a known size to ensure accurate measurements and/oraccurate viewing of the images. Additionally, the overlaid images can becalibrated to the surgical site itself. The calibration can involveemploying a calibration piece, e.g., a ruler or grid, and placing thecalibration piece at the location of the surgical site and imaging thecalibration piece to obtain the proper image size for the fiducial. Animage of the fiducial of proper size can then be stored and the storedimage of the fiducial can be reproduced on the second display. Properadjustment of the scaling of the first and second display, to match eachother may be undertaken as well. Other approaches to calibration mayalso be employed. The calibration can utilize predetermined distancesautomatically applied or applied through user input. The calibration canbe performed electronically by electronics that identifies the image ofthe surgical field, the surgical field itself, and/or an aspect of theimage or surgical field and provides scaling or magnificationaccordingly. Additionally, input controls can be utilized by the user tocalibrate or scale the images to match each other in size or match thescale of the surgical field itself or features therein. For example, auser (e.g., a technician) can manipulate the displays by moving them inany direction increasing or decreasing the magnification so that theimage fields of the display(s) are lined up and scaled to the correctsize. This can be done by utilizing knobs, buttons, touch screens,virtual touch screens, voice or other input controls from a control unitand manipulated by the user. The control unit can be integrated into thebinocular display assembly. In other embodiments, the control unit canbe remote from the binocular display assembly.

In some embodiments, an image can be used to display drawings or othermarkings drawn by or inserted by the surgeon or assistant on thesurgical field. The drawing(s) and other marking(s) can be superimposedon or displayed adjacent to a display of an image of the surgical fieldor projected adjacent to and/or on the surgical field. For example, asurgeon can introduce a drawing and/or annotation onto an image of thesurgical field to annotate the image as desired. In some embodiments,the drawings or annotations can be projected onto the surgical field oronto the patient. In some embodiments, a computer processor receives thedrawings or annotation produced or selected by the surgeon and identifywhere the annotations or drawings are being inserted relative to thedisplay of the surgical field image or relative to the surgical fielditself. In some embodiments, the computer processor can save and/orreuse these drawings or annotations. In various embodiments, thedrawings and/or annotation can also appear on one or more assistantdisplays, and/or on a panel display viewable by others in the room or byone or more displays located remotely.

In some embodiments, an image can be used to simulate placement ofimplants in the surgical field. The image of the implant can allow forthe implant to be aligned virtually with an image of the surgical fieldand/or the surgical field. For example, an overlay of a spinal implantcan be used to illustrate and/or determine the desired positioning ofthe spinal implant within the surgical area. In some embodiments, thespinal implant image can be manipulated by the user to move the implantimage over the surgical area or surgical field display. In variousembodiments, a variety of implant images can be stored or retrievedbased on user selection of the type of implant. In some embodiments,input controls can be utilized by the user to move the image of thesurgical implant relative to the surgical field. In some embodiments,the movement of the image of the surgical implant can be performed by aninput of the user to move the image of the surgical implant in anydirection as desired by the user. For example, the user can manipulatethe image displays by moving them in any direction. This can be done byutilizing knobs, buttons, touch screens, virtual systems and methods,voice, or other input controls from a control unit manipulated by theuser. The control unit can be integrated into the binocular displayassembly. In other embodiments, the control unit can be remote from thebinocular display assembly. In some embodiments, the image of theimplant can be magnified or scaled to the appropriate size by a computerprocessor or by other mechanical control or inputs as describedpreviously. Alternatively, the implant image can be displayed at apredetermined size corresponding to the actual size of the implant. Invarious embodiments, the image of the implant can also appear on one ormore assistant displays, and/or on a panel display viewable by others inthe room or by one or more displays located remotely.

Illumination

Various implementations described herein can include an imaging systemto generate images of a surgical site. The imaging system can include acommon objective for left and right eye optical paths. The commonobjective can collimate light from the surgical site. The optical pathof the collected collimated light can be redirected by a prism, foldelement, or beamsplitter. The collimated light can be configured to passthrough a zoom lens group (e.g. afocal zoom), an aperture, and otheroptical elements to direct portions of the collimated light within theleft and right eye optical paths, as shown in FIG. 43 described hereinand/or FIGS. 29A and 29B of U.S. application Ser. No. 14/581,779 filedon Dec. 23, 2014 entitled “SURGICAL VISULAIZATION SYSTEMS,” whichpublished as U.S. Publication No. 2015/0297311. U.S. application Ser.No. 14/581,779 and U.S. Publication No. 2015/0297311 is incorporated byreference herein in its entirety for all that it discloses. For eachleft and right eye optical paths, the imaging system can be configuredto generate images of the surgical site. The imaging system can includea sensing system configured to detect the images formed in the left andright eye optical paths. For example, in some embodiments, the sensingsystem can be configured to detect the images formed in the left eyeoptical path on portion of the sensing system, and to detect thegenerated images in the right eye optical path on another portion of thesensing system.

Each left and right eye optical paths of the imaging system can includea plurality of optical paths for imaging at different wavelength ranges.In some embodiments, the optical paths can lead to sensors sensitive tothe different wavelength ranges. In various embodiments, the pluralityof optical paths can be used for imaging, for example, in infrared (IR),near infrared (NIR), visible wavelengths or bands (about 400 to about700 nm). As described herein, features (e.g., tumors) could be made tofluoresce by injecting fluorescent chemical (e.g., dye) and illuminatingthe feature with light that induces fluorescence (e.g., UV, visible, orin IR radiation). As another example, imaging at blue (about 440 toabout 460 nm) and/or green (about 540 to about 560 nm) wavelengths canimprove visibility of features since the peak light absorption ofhemoglobin occurs at these narrow band wavelengths. Use of other bandsof wavelengths are possible.

In certain embodiments, the imaging system can switch between imaging atdifferent wavelengths. For example, in some embodiments, the imagingsystem can switch between imaging at narrow band wavelengths and visiblewavelengths. As another example, in some embodiments, the imaging systemcan switch between imaging at NIR and IR. Thus, certain embodiments canswitch between wavelengths or wavelength bands in two or more or all ofinfrared (IR), near infrared (NIR), visible, and ultraviolet (UV). Byswitching between different wavelength ranges (e.g., illuminating thesurgical site with different wavelength ranges in sequence), the surgeonor assistant may be able to detect subtle differences between the twoimages. If the images are registered, the surgeon would be able to seeand know a sense of direction. Also, by switching illumination on andoff, the sensors can remain on and detect the appropriate wavelengthswhen present based on the illumination that is switched on. As anexample, certain embodiments can include a sensor sensitive to visiblelight and another sensor sensitive to IR light. When the visible lightsource is on, the sensor sensitive to visible light can detect thevisible light, and when the IR light source is on, the sensor sensitiveto IR light can detect the IR light. Furthermore, some embodiments caninclude a single sensor with multiple illumination (e.g., dual visibleillumination).

In some such embodiments, the imaging system can include an illuminationcontrol configured to control the light provided to the surgical sitesuch that the provided light switches between the light of differentwavelength ranges. For example, the surgical site can be illuminated byat least one illumination source, such as one or more lasers, lightemitting diodes, or white-light light sources. The illumination sourcecan include multi-spectral illumination sources to provide visible lightor white light whose color temperature can be varied. The illuminationsource can also include other sources configured to provide infrared,near infrared, visible, ultraviolet, and or other bands of wavelengths.The illumination control can control the light provided to the surgicalsite by controlling one or more illumination sources. For example, theillumination control can adjust the position of an illumination source(e.g., position of a fiber optic) and/or turn on/off an illuminationsource or block the illumination (e.g., with a shutter or modulator). Asanother example, the illumination control can include a filter (e.g., abandpass, low pass, or high pass filter) such that the sensors areprovided light of the different wavelength ranges. The illuminationcontrol can include at least one knob, dial, button, handle, switch,joystick, haptic, or touchpad. In some embodiments, the illuminationcontrol can be disposed on the binocular viewing assembly including ahousing and oculars such as on handles on the housing such that the userneed not remove one's eyes from the oculars.

Alternatively, instead of controlling the light being provided to thesurgical site, some embodiments of imaging systems can control thereceived light from the surgical site. For example, each of the left eyeand right eye optical paths can include one or more filters (e.g., anotch filter) configured to remove unwanted wavelengths from thereceived light to remove unwanted wavelengths from the light beingprovided to the surgical site. For example, a notch filter may be usedin front of the different sensors to block the pump light uses to excitefluorescence, which may possibly saturate the sensors. The differentwavelengths ranges can include wavelengths in two or more or all ofinfrared (IR), near infrared (NIR), visible, and/or ultraviolet (UV), asdescribed herein.

Various embodiments described herein include an imaging systemconfigured to generate images of the surgical site at differentwavelength ranges. The imaging systems can be used for the primarydisplay, surgeon display, the assistant display, possibly otherdisplays, or any combination of these. For example, in some embodiments,the assistant can switch to and from a fluorescence image on theassistant display. In some cases, the assistant can switch the imagesthat the assistant sees from fluorescence images to visible images andback or combinations of thereof, without changing the images seen by theprimary surgeon. In various embodiments described herein, the generatedimages in the left eye and right eye optical paths can be differentstereo views of the surgical field. In some embodiments, the imagingsystem can provide a surgical microscope view. As for variousembodiments described herein, the imaging system need not be associatedwith a direct view surgical microscope.

Additional Discussion on Producing Multiple Images in DifferentWavelength Bands

Accordingly, in various embodiments, one or more cameras, such as thecamera that provides a surgical microscope view (e.g. having a workingdistance and/or focal length of 150-450 mm) can be configured to receiveimages of different wavelengths. The cameras may, for example, receivefluorescence images as well as non-fluorescence images or narrow bandimages such as blue images (e.g., about 440 to about 460 nm wavelength)and/or green image (about 540 to about 560 nm wavelength) as well asbroad band visible wavelength images (e.g., that extend across thevisible spectrum) wherein the surgical site is illuminated with whitelight. Such a visible wavelength image may be used as a reference viewused by the surgeon most of the time in performing the surgery. Thisview may resemble the view seen through a direct view surgicalmicroscope.

Likewise, the cameras may have sensors that are sensitive to differentwavelength bands. For example, a visible detector array and an infrareddetector array (e.g., NIR) may be employed. Or different types ofsensors sensitive to different portions of the infrared spectrum may beused. Similarly, different types of sensors sensitive to differentportions of the visible spectrum may be used.

Alternatively (or in combination), multiple sensors having similarspectral response can be used. Additional wavelength selective opticsmay be used to enable the different sensors to detect differentwavelengths. For example, multiple visible detectors with the samespectral responsivity can be used. Similarly, multiple infrareddetectors with the same spectral responsivity may be use. One or morewavelength selective filters may be associated with one or more of thesensors and may be included in the optical path to the respectivesensors. For example, a first filter or filter combination may beincluded in the optical path to a first sensor and a second differentfilter or filter combination may be included in the optical path to asecond sensor. In this manner, the two sensors may be sensitive to twodifferent spectral bands, e.g., blue and green respectively. In somecases, one of the sensors may not have any filters associated therewithor may otherwise be configured to be sensitive to a broader wavelengthspectrum that is different from the other optical sensor. For example, afirst filter or filter combination may be included in the optical pathto a first sensor and while no filter is included in the optical path toa second sensor. Again, the two sensors will be sensitive to twodifferent spectral bands, e.g., green and the entire visible spectrumrespectively. This later configuration may be useful, for example, tosee a fluorescence peak on a first sensor and a broad band image (e.g.,reference view) across the entire visible spectrum that is produced byilluminating the surgical site with white light on a second sensor.

Accordingly, one or more of the sensors may have associated therewith afilter or filter combination to provide wavelength selectivity. Thedifferent filters for the sensors may comprise band pass filters, lowpass filters, high pass filters, notch filters, or any combinationthereof.

In some embodiments, the wavelength selectivity may be included in oneor more beamsplitting element (e.g., a beamsplitter such as a dichroicbeamsplitter). Such an optical element may enable the separation oflight having two different spectra into two separate beams. For example,light of having a first spectra may be directed (e.g., reflected) in afirst direction while light having another different second spectra maypass through. Accordingly, additional filters need not be added in theoptical paths of the sensors although in some embodiments suchadditional filters may be included. Use of wavelength selectivebeamsplitters or deflectors can provide increased efficiency incomparison to a 50:50 beamsplitter with filters to remove unwantedlight. Instead of removing unwanted wavelengths, such as by usingabsorption filters, the wavelengths are redirected. However, in variousembodiments that employ wavelength selective beamsplitters ordeflectors, filters may still be employed to further tailor thespectrum.

Additionally, as described herein switching illumination on and off mayprovide the wavelength selectivity. For example, a first light sourcehaving a first wavelength spectra may be turned on while a second lightsource having a second spectra is shut off or blocked from reaching thesurgical site at a first time. The light or resulting fluorescenceresponse signal reaches the one or more sensors to provide an imagecorresponding to the first wavelength spectra. At a second later time,the second light source is turned on or permitted to reach the surgicalsite while the first light source is turned off or blocked from reachingthe surgical site. This light then reaches the one or more sensors toprovide an image corresponding to the second wavelength spectra.

In some such embodiments, a single sensor may be used to collect lightfrom either light source if the sensor is sensitive to both lightsources. In other embodiments multiple sensors may be employed. Onesensor may, for example, be more sensitive to the first wavelengthspectra and another sensor may be more sensitive to the secondwavelength sensor. One or more wavelength selective beamsplitter ordeflector may be used to direct the light of the different wavelengthsto the different sensors.

Instead of switching the light sources on and off, in certainembodiments both light sources can remain on and the images fromrespective sensors can be sent to different displays. Or the viewer canswitch from showing the image from the first sensor on the display toshowing the image from the second sensor on the display and vice versa.Accordingly, the viewer can switch back and forth between views of thesurgical site having different wavelength distributions. Switching backand forth may enable the viewer to more readily detect distinctionsbetween the two images.

Alternatively, in certain embodiments, the viewer can view both imagesfrom both cameras at the same time, for example, in a picture in picture(PIP) configuration or next to each other, etc. As described herein, thesurgical visualization system may be configured for the viewer to selectwhich format to view the images, for example, one at a time with abilityto switch from one image to the other, or both at the same timejuxtaposed on the display.

The visualization system may be configured to provide the primarysurgeon these different images in different spectral images either atthe same time or to switch between them. The surgeon can control whatimage he or she sees.

Similarly, the visualization system may be configured to provide theassistant or other viewer besides the primary surgeon the ability toview the different images in different spectral either at the same timeor to switch between them. The assistant or other viewer can controlwhat image he or she sees regardless of what views or images thatprimary surgeon elects to see. For example, the assistant may beinterested in seeing only a fluorescence image or only a narrow bandblue or green image while the primary surgeon is interested in switchingback and forth between the fluorescence image or a narrow band blue orgreen images and a broad band visible image covering the width of thevisible spectrum.

In some embodiments, the illumination directed onto the surgical sitepasses through the imaging optics for receiving light from the surgicalsite before being incident on the surgical site. For example, the lightsource is disposed with respect to the imaging optics (or a portionthereof) to form a path from the light source through the imaging optics(or a portion thereof) to the surgical site. This light may first thenpass through, for example the objective and/or the zoom system prior tobeing incident on the surgical site. A portion of this light after beingreflected or a fluorescence response signal from the surgical site willthen pass again through the imaging optics and be directed to the one ormore sensors.

Additionally, in various embodiments more than two optical sensors maybe employed. For example, more than two detector arrays for the leftchannel in a stereo camera may be employed. Likewise, more than twodetector arrays for the right channel in a stereo camera may beemployed. As discussed above, multiple sensors may increase the spectralinformation provided by the system as many spectral regions may beseparately measured and be available to the viewer for comparison.

In certain embodiments, for example, 4 cameras sending signal to 4displays (e.g., 2 cameras and displays for each of left and rightchannels). Alternatively, 6 cameras could send signal to 6 displays(e.g., 3 cameras and displays for each of left and right channels).Alternatively, 8 cameras could send signal to 8 displays (e.g., 4cameras and displays for each of left and right channels). The number ofcameras and displays may be more or less. Additionally, mono cameras (ascompared to stereo cameras) may be included. The number of cameras anddisplays also need not be the same. Similarly, the number of camera'sand/or displays associated with the left channel need not be identicalto the number of camera's and/or displays in the right channel.

Various combination of the features described above and elsewhere hereinmay be used.

Example Embodiments on Producing Multiple Images in Different WavelengthBands

FIGS. 32 and 33 show example embodiments of a combination of abeamsplitter and one or more filters which can be used in an imagingsystem as described herein to generate images of a surgical site indifferent wavelength ranges. In certain embodiments, the imaging systemcan include a filter block (or a filter cube, cartridge, or assembly) tohold a beamsplitter and one or more filters. For example, as shown inFIG. 34, the filter block or assembly can be placed in the optical pathfrom the surgical site to the sensing system. In particular, the filterassembly comprising the dichroic beamsplitter is shown in the opticalpath between the focusing optics and the sensing system. The filterblock can be moved in and out of the imaging system allowing thebeamsplitter and one or more filters to be changed or allowing thefilter block comprising the beamsplitter and one or more filters to bereplaced with another filter block comprising a beamsplitter and one ormore filters.

As shown in FIG. 32, the filter block, cartridge, or assembly caninclude an excitation blocking filter, a dichroic beamsplitter, and aclean-up filter. In this example, one or more light sources can providevisible light and near infrared (NIR) light to illuminate the surgicalsite. This visible and NIR may include pump light for excitingfluorescence. Optical fluorescence in the form of NIR light in thisexample may be emitted from the surgical site. As described herein, anexcitation blocking filter can be used in front of the sensing system toblock the pump light used to excite fluorescence, which may possiblysaturate the sensing system. The dichroic beamsplitter (used alone or incombination with one or more filters) can separate the visible light andNIR light into two optical paths. In this example, a clean-up filter isused in one of the optical paths. Such a filter may, for example, reducecontribution from wavelengths different from the fluorescence signal.The sensing system, which may include one or more sensors in eachoptical path, can be used to detect the separated visible and/or NIRfluorescence light. The imaging system can generate the images of thesurgical site based on the detected light. For example, the imagingsystem can generate an image of the surgical site in the visible range,and another image of the surgical site in the near infrared range. Thisvisible image may be a useful reference view which the surgeon uses formost of the surgery and provides a visible image akin to what thesurgeon is accustomed to when viewing a patient through a surgicalmicroscope with a white illumination light source.

As shown in FIG. 33, one or more light sources can be used to illuminatethe surgical site with visible and near infrared light. This visible andNIR may include pump light for exciting fluorescence. Opticalfluorescence in the form of visible light in this example may be emittedfrom the surgical site. The dichroic beamsplitter can separate the lightinto two optical paths with one or more filters in each optical path. Inthis example, in one optical path, a NIR blocking filter is used toblock the NIR light such that only the visible light is sent to thefirst optical path. In the other optical path, a clean-up filter is usedsuch that mostly only or only the visible fluorescence light is sent tothe other optical path. The sensing system, which may include one ormore sensors, can be used to detect the light in each of the two paths.

Advantageously one filter block or assembly having first spectralcharacteristics can be switched out (e.g., by a nurse, technician,orderly, doctor or surgical staff) for a second filter block or assemblyhaving second spectral characteristics. Accordingly, different filterassemblies can be used for different circumstances, for example, ifdifferent fluorescence dyes and fluorescent peaks are used such as fordifferent procedures. In some embodiments, the filter block or assemblymay simply comprise a dichroic or wavelength selective beamsplitter. Inother embodiments, the filter block or assembly may comprise filterssuch as band pass filters and/or notch filters. As described above, onefilter assembly can be switched out for another filter assembly fordifferent circumstances. For example, the dichroic beamsplitters in FIG.34 can be switched out for other dichroic beamsplitters having differentspectral properties such that although the first dichroic filter mightsplit light into first and second beams having first and second spectraldistributions, the second dichroic filter might split light into firstand second beams having different third and four spectral distributions.An opening or door in the side or top or bottom of the housing of thecamera may permit the filter assembly to be switched out.

Other combinations of light sources, beamsplitters, filters, andwavelength bands are possible. As described herein, in variousembodiments, the imaging system can generate images in differentwavelength bands. The imaging system can combine (e.g., dispose asadjacent to one another, tile, overlap, superimpose, dispose as PIP,etc.) and/or switch between the images.

Fluorescence and Narrow Band Imaging

FIG. 35 (lower portion) illustrates various embodiments where thesurgical microscope view camera obtains a fluorescence image as well asa visible reference image. Light collected by the microscope objectiveis passed through a dichroic beamsplitter and directed into two paths.The first path (channel 1) for light passing through the beamsplitter isfor the fluoresce signal. A sensor is included in channel 1 that issensitive to the fluorescent signal. The second path (channel 2),reflected by the beamsplitter, is for a broad band visible wavelengthreference view. The broad band image might resemble an image seen by asurgeon when viewing a patient illuminated with white light through adirect view surgical microscope. A sensor is included in channel 2 thatis sensitive to visible light. The wavelength selective beamsplitter maybe used to separate the light into the two paths. A blocking filterconfigured to block the pump wavelength used to induce fluorescence isalso included.

As discussed elsewhere herein the dichroic beamsplitter may beinterchangeable. For example, one dichroic beamsplitter can be switchedout for another if a different fluorescence wavelength is to be examinedor a different type of image or images are to be obtained. For example,different procedures using different fluorescing materials such as dyesmay result in different fluorescent wavelengths. Accordingly, thewavelength selective beamsplitter may be different. An opening in thehousing for easy substitution of one dichroic filter for another may beused. Similarly, the blocking filter may be switched out, for example,if the pump wavelength is different. The blocking filter and dichroicbeamsplitter may be included in a single assembly that is switched outin certain embodiments.

FIG. 35 (upper portion) also shows a display assembly such as abinocular display assembly that includes multiple optical channels(channel 1 and channel 2) and associated display screens (e.g., LCD,OLED). A beam combiner combines the optical paths to each of the displayscreens. One of the display screens (channel 1) can be in electricalcommunication with the sensor receiving the fluorescence signal. Theother display screen (channel 2) can be in electrical communication withthe sensor receiving the visible reference image.

FIG. 36 (lower portion) illustrates various embodiments configured fornarrow band imaging, for example, using blue and green wavelengths.These wavelengths may, for example, be between 440 to 460 nm and 540 to560 nm, respectively. FIG. 36 shows the surgical microscope view cameraconfigured to obtain narrow band images as well as visible referenceimages. Light collected by the microscope objective is passed through adichroic beamsplitter and directed into two paths. The first path(channel 1) for light passing through the beamsplitter is for the narrowband imaging signal. A sensor is included in channel 1 that is sensitiveto the narrow band imaging signal. The second path (channel 2), reflectby the beamsplitter is for a broad band visible wavelength referenceview. The broad band image might resemble an image seen by a surgeonwhen viewing a patient illuminated with white light through a directview surgical microscope. A sensor is included in channel 2 that issensitive to visible light. The wavelength selective beamsplitter may beused to separate the light into the two paths. A narrow band imagingfilter configured to further attenuate unwanted wavelengths, forexample, outside 440 to 460 nm and 540 to 560 nm, is also included.

As discussed elsewhere herein the dichroic beamsplitter may beinterchangeable. For example, one dichroic beamsplitter useful fornarrow band imaging can be switched out for another useful forfluorescence imaging. Accordingly, the wavelength selective beamsplittermay be different. An opening in the housing for easy substitution of onedichroic filter for another may be used. Similarly, the filter may beswitched out through an opening. The filter and dichroic beamsplittermay be included in a single assembly that is switched out in certainembodiments.

In certain embodiments, a beamsplitter that is not a dichroicbeamsplitter may be used. The narrow band imaging filter may be used tofilter our light outside the narrow band imaging wavelengths toaccomplish narrow band imaging.

FIG. 36 (upper portion) also shows a display assembly such as abinocular display assembly that includes multiple optical channels(channel 1 and channel 2) and associated display screens (e.g., LCD,OLED). A beam combiner combines the optical path to each of the displayscreens. One of the display screens (channel 1) can be in electricalcommunication with the sensor receiving the narrow band imaging signal.The other display screens (channel 2) can be in electrical communicationwith the sensor receiving the visible reference image.

A wide range of variations are possible. Various features may beexcluded and/or combined with other features disclosed elsewhere herein.

Binocular Viewing Assembly

The disclosure generally describes a binocular viewing assembly forproviding a surgeon with video of a surgical site. The binocular displaycan include a plurality of displays in a housing and a plurality ofoculars for viewing those displays. The binocular viewing assembly canbe configured to show views of video images of the surgical site. Thebinocular viewing assembly can be further configured to show images fromthe surgeon's cell phone or tablet. In some embodiments, the binocularviewing assembly includes optical components configured to direct videoimages from the plurality of displays to the oculars, wherein the videoimages are acquired with stereo electronic microscope cameras configuredto provide a surgical microscope view of the surgical site. The opticalcomponents can be configured so that the optical axis from the ocularsis not aligned with (e.g., does not intersect) the stereo electronicmicroscope cameras.

In some embodiments, the binocular viewing assembly can be combined withone or more displays positioned outside of the housing, wherein thedisplay can be viewed by an assistant or other personnel in theoperating room. The one or more displays can be flat panel displays orimages projected onto a screen or other surface so that a person notlooking through the oculars of the binocular viewing assembly can seevideo images on the one or more displays. In certain implementations,the video images on at least one of the one or more displays can includethe view that is being provided in the binocular viewing assembly. Insome implementations, a plurality of displays can be configured todisplay the video images provided by the plurality of displays in thebinocular viewing assembly. In certain embodiments, the displays aremounted on the housing. In some embodiments, one or more of the displayscan be configured to provide stereo video images. In some embodiments,the binocular viewing assembly can include a fiber optic light source todirect light to the surgical site. In some embodiments, the binocularviewing assembly can be configured to provide views of video acquiredwith one or more endoscopes.

In some embodiments, the surgical visualization system can include oneor more endoscopes, one or more cameras positioned on a retractor and/orone or more cameras positioned proximal to the surgical site in additionto the stereo microscope camera. The various cameras can have differentoptical properties to provide different imaging functionality to thesurgical visualization system. For example, the one or more cameraspositioned on the retractor can have a relatively wide field of view andcan be positioned at a distance that is multiple focal lengths from thesurgical site. The one or more cameras on the retractor can thus beconfigured to provide a relatively short focal length and wide field ofview, to provide images suitable for working more obliquely. As anotherexample, the stereo microscope camera can be configured to have a longerfocal length than the cameras on the retractor and/or the proximalcameras. The stereo microscope camera can be configured to be positionedabout one focal length away from the surgical site. The stereo cameracan be configured to provide zooming functionality. The stereomicroscope camera can be positioned to allow a surgeon to put toolsand/or proximal cameras between the surgical microscope camera and thesurgical site. The stereo microscope camera can have a relatively narrowfield of view. In certain implementations, the proximal cameras can bepositioned on a structure so that the proximal cameras are positionedoutside of the surgical site and below the stereo microscope camera andthe binocular viewing assembly, wherein the structure is positioned sothat a surgeon can put tools between the structure and the stereomicroscope camera.

In certain implementations, the surgical visualization system caninclude one or more projectors configured to project images on apatient. The images can be of user interface elements to allow virtualmanipulation by a surgeon utilizing the binocular viewing assembly. Thesurgical visualization system can include a user interface cameraconfigured to acquire video images of the patient and the hands of thesurgeon with the projected images. The user interface camera can beoperably coupled to an image processing system to determine whethermovements by the surgeon's hands correspond to virtual manipulation ofthe user interface elements of the projected images.

The disclosure also provides for a binocular viewing display havingoculars beneath corresponding displays. The binocular viewing displaycan be configured in a manner similar to a periscope. For example, thedisplays can be positioned above the oculars with optical componentsconfigured to deliver images of the displays positioned underneath thedisplays.

The disclosure also provides for a binocular display assembly comprisinga contoured housing. The contoured housing can include, for example, atleast an indentation on a bottom portion of the housing, wherein theindentation is configured to provide a place to attach or otherwiseposition a camera for acquiring images of a surgical site.

The disclosure also provides for a surgical visualization system thatincludes a primary surgeon camera and an assistant camera, the primarysurgeon camera and the assistant camera positioned to acquire images ofa surgical site from outside the surgical site. The surgicalvisualization system can include an optical system associated with theprimary surgeon camera and the assistant camera, wherein the opticalsystem includes a central objective that is rotatable around a centralaperture. The primary surgeon camera and the assistant camera areconfigured to rotate about the central aperture that encircles thecommon objective or pairs of converging optical trains.

The disclosure also provides for a microscope head that can be used withone or more of the surgical visualization systems disclosed herein,wherein the microscope head is autoclavable. For example, the microscopehead can be made of materials that can withstand heats and pressurespresent in an autoclave used to sterilize devices and/or objects for usein surgery. The materials can include polymers such as, for example andwithout limitation, polypropylene, polymethylpentene, PTFE resin,polycarbonate, polymethyl methacrylate, and the like. Other materialsinclude, for example and without limitation, metals, plastics, rubber,and other such materials or combinations of materials.

Example Binocular Viewing Assembly

FIGS. 37A-B illustrate an example binocular viewing assembly 3700 forproviding a surgeon 3720 with video of a surgical site 3730. Thebinocular viewing assembly 3700 can include a housing 3705 attached to asupport structure 3702 through arm 3703 a that allows a position and/ororientation of the binocular viewing assembly 3700 to be manipulated.The binocular viewing assembly 3700 includes oculars 3707 configured toprovide a view of a binocular display positioned in the housing 3705.The binocular display can comprise one or more electronic displays. Thebinocular display can comprise one or more electronic displays for eachof a left eye and a right eye optical viewing path from the oculars3707. The binocular viewing assembly 3700 includes a stereo surgicalmicroscope camera assembly 3710 coupled to the support structure 3702through arm 3703 b. The housing 3705 can include handles 3709 tofacilitate manipulation of the housing 3705. The arm 3703 b can allowfor independent manipulation of the position and/or orientation of thestereo surgical microscope camera assembly 3710 (e.g., independent ofthe movement, position, and/or orientation of the housing 3705).

The stereo surgical microscope camera assembly 3710 can include a stereomicroscope camera configured to acquire video images of a surgical siteor other work site. The stereo microscope camera can be configured toprovide a surgical microscope view of the surgical site. The stereomicroscope camera can be configured to be positioned from the surgicalsite at least about 150 mm and/or less than or equal to about 450 mm, atleast about 175 mm and/or less than or equal to about 400 mm, or atleast about 200 mm and/or less than or equal to about 300 mm. Byproviding these ranges of working distance, this can allow the surgeonto place tools and/or other cameras between the stereo surgicalmicroscope camera assembly 3710 and the surgical site. The stereomicroscope camera can be configured to have a focal length that is aboutthe same as the working distance. The stereo microscope camera can beconfigured to provide a zoom functionality to allow for magnification ofvideo images of the surgical site. For example, at a fixed workingdistance the stereo microscope camera can be configured to zoom in andout of the surgical site to allow magnification of different portions ofthe surgical site. The stereo microscope camera can be configured tohave a relatively narrow field of view. For example, where the workingdistance and/or focal length of the stereo microscope camera is about300 mm, the field of view of the stereo microscope camera can allowvisualization of an area of about 30 mm in diameter. The field of viewof the stereo microscope camera can be between, for example and withoutlimitation, about 50 mm and about 100 mm. The field of view of thestereo microscope camera can be, for example and without limitation, atleast about 5 degrees and/or less than or equal to about 15 degrees.

In some embodiments, the binocular viewing assembly 3700 provides a viewof one or more additional cameras. For example, one or more cameras canbe positioned on an endoscope or a retractor and at least one of theplurality of displays of the binocular viewing assembly 3700 can beconfigured to display images from the one or more cameras on theendoscope or retractor. The one or more cameras on the endoscope orretractor can be configured to have a relatively short focal distanceand/or a relatively wide field of view, compared to the stereomicroscope camera. For example, a camera positioned on the endoscope orretractor can have a focal length of about 3 mm. The one or more cameraspositioned on the endoscope or retractor may also be positioned multiplefocal lengths from the portion of the surgical site to be imaged. Forexample, where the focal length of an endoscope or retractor camera isabout 3 mm, the camera can be positioned about 10 mm from the patient orportion of the patient to be imaged. Thus, the endoscope or retractorcameras can be in relatively close proximity to the object to be imagedand provide a wide field of view.

In some embodiments, the binocular viewing assembly 3700 provides a viewof one or more additional cameras positioned on a standoff structure,positioned proximal to the worksite or surgical site. These proximalcameras can be configured to provide images of a human body or partthereof from outside of the surgical site, but from a relatively closedistance thereto. For example, the standoff structure and/or thepositioning of the proximal cameras can be such that a surgeon cannoteasily put a tool or their hands between the structure (or the cameras)and the surgical site so that the surgeon can work freely in that space.

The binocular viewing assembly 3700 can be configured to show views ofvideo images of the surgical site. The video images can be acquired withthe stereo microscope camera, endoscope cameras, retractor cameras,proximal cameras, or any combination of these. The plurality of displayscan be configured to provide images acquired with a particular camera.For example, one display can be configured to provide images from asingle camera (e.g., a stereo camera). In certain implementations, theoculars 3707 provide views of a plurality of displays, each displayconfigured to display video images acquired by a particular camera. Insome embodiments, the oculars are configured to combine first and secondvideo images at the oculars from first and second displays to provide acombined image to the viewer. This can be done to optically superimposeimages with a relatively small latency between image acquisition andimage display. This can be beneficial for a surgeon due to rapid ornearly instantaneous visual feedback between actions and what is viewedthrough the oculars 3707.

In some embodiments, the optical axis from the oculars 3707 does notform a straight line to the stereo electronic microscope cameras toprovide a surgical microscope view. For example, the stereo electronicmicroscope camera can be positioned below the line of sight of theoculars 3707 and be directed at different pitch angles (and/or yawangles) and possibly be offset in x, y, and/or z.

FIG. 37B illustrates the surgical microscope camera assembly 3710configured for providing a surgical microscope view of a surgical site3730 from a temporal approach. The surgical microscope camera assembly3710 can be rotated and positioned to acquire video images of thesurgical site 3730 wherein the optical axis from the surgical microscopecamera assembly 3710 to the surgical site is substantially horizontal.In some embodiments, the optical axis can be inclined with respect tothe horizon. In certain implementations, the optical axis issubstantially orthogonal to the surgical site.

When configured for use with a temporal approach, the surgicalmicroscope camera assembly 3710 can be configured to be positionedsubstantially below the housing 3705. In certain implementations, thesurgical microscope camera assembly 3710 can be positioned between asurgeon's hands and/or arms during surgery. The surgical microscopecamera assembly 3710 can be configured to be relatively small so thatthe surgical microscope camera assembly 3710 does not significantlyimpede with free movement of the surgeon's hands, arms, and/or tools. Incertain implementations, the surgeon can use tools 3731 that arerelatively long (e.g., between about 150 mm and about 300 mm). Thus, thesurgical microscope camera assembly 3710 can be positioned to provide aworking distance between about 150 mm and about 450 mm when positionedbetween the surgeon's hands or arms and long tools are utilized.

The surgical microscope camera assembly 3710 can be configured to besmaller than the binocular viewing assembly 3700. For example, thesurgical microscope camera assembly 3710 can have a linear dimension ofless than or equal to about 6 inches and at least 4 inches or less thanor equal to about 140 mm and at least about 65 mm. In particular, someembodiments of the surgical microscope camera assembly 3710 include ahousing with a depth of about 5.5 inches, a height of about 5.2 inches,and a width of about 4 inches or a depth of about 140 mm, a height ofabout 100 mm (or about 65 mm for a lens assembly), and a width of about100 mm, as illustrated in FIG. 44. The dimensions of surgical microscopecamera assembly 3710 can be larger or smaller. For example, a height ofthe surgical microscope camera assembly 3710 housing can be less than orequal to about 9 inches, less than or equal to about 8 inches, less thanor equal to about 7 inches, less than or equal to about 6 inches, lessthan or equal to about 5 inches, less than or equal to about 4 inches,or less than or equal to about 3 inches or any range between any ofthese values. For example, a depth of the surgical microscope cameraassembly 3710 housing can be less than or equal to about 9 inches, lessthan or equal to about 8 inches, less than or equal to about 7 inches,less than or equal to about 6 inches, less than or equal to about 5inches, less than or equal to about 4 inches, or less than or equal toabout 3 inches or any range between any of these values. For example, awidth of the surgical microscope camera assembly 3710 housing can beless than or equal to about 9 inches, less than or equal to about 8inches, less than or equal to about 7 inches, less than or equal toabout 6 inches, less than or equal to about 5 inches, less than or equalto about 4 inches, or less than or equal to about 3 inches or any rangebetween any of these values. As another example a volume of the surgicalmicroscope camera assembly 3710 housing can be less than or equal toabout 730 in³, less than or equal to about 525 in³, less than or equalto about 350 in³, less than or equal to about 225 in³, less than orequal to about 115 in³, less than or equal to about 100 in³, less thanor equal to about 75 in³, or less than or equal to about 50 in³ or anyrange between any of these values. The surgical microscope cameraassembly 3710 can weigh less than or equal to about 7 kg, less than orequal to about 6 kg, less than or equal to about 5 kg, less than orequal to about 4 kg, less than or equal to about 3 kg, less than orequal to about 2 kg, or any range between any of these values. Thesurgical microscope camera assembly 3710 with the housing conforming tothe above size restrictions can be configured to capture video imageswith a resolution of at least 1920 pixels by 1080 pixels (e.g., “HD”video). The surgical microscope camera assembly 3710 with the housingconforming to the above size restrictions can be configured to have aworking distance of at least 150 mm and/or less than or equal to about450 mm, at least 200 mm and/or less than or equal to about 400 mm, atleast 250 mm and/or less than or equal to about 350 mm, or at least 275mm and less than or equal to about 325 mm. When the surgical microscopecamera assembly 3710 is thus configured in a compact housing, thesurgical microscope camera assembly 3710 can be positioned near thebinocular viewing assembly 3700 and/or near the surgeon 3720 withoutsignificantly impeding free movement of the surgeon's hands and/or toolsduring surgery (e.g., when the surgeon is operating and looking throughthe oculars 3707 of the binocular viewing assembly 3700).

In some embodiments, the binocular viewing assembly 3700 can beconfigured such that the surgeon 3720 can look at the surgical site 3730(e.g., directly look at the surgical site without the use of thebinocular viewing assembly 3700) without visual obstructions from thebinocular viewing assembly 3700. In various embodiments, the surgeon3720 can switch between looking through the oculars 3707 and directlylooking at the surgical site 3730 without having to move components ofthe binocular viewing assembly 3700 and/or without having tosignificantly move from their body (e.g., their hands and arms canremain relatively stationary, which can be important when performingsurgery).

In some embodiments, the binocular viewing assembly 3700 is configuredfor use with a tube access approach. For example, where tools are usedin combination with a tube access approach, the surgeon's tools can beconfigured to push the surgeon's hands away from the tube opening toallow the surgeon to view down the tube. With the binocular viewingassembly 3700, the surgical microscope camera assembly 3710 can beconfigured to provide the view through the tube. For example, thesurgical microscope camera assembly 3710 can include an objective lenspositioned about 100 mm from the tube opening and configured to have anoptical axis through the tube to the surgical site 3730.

In some embodiments, the binocular viewing assembly 3700 can be furtherconfigured to show images from a cell phone, tablet, or other computingdevice, such as, for example, described above. In certain embodiments,the computing device can be placed within or near the housing 3705 sothat the user can see the display of the computing device. For example,an optical system can be provided that relays an image of the computingdevice to the oculars 3707. In certain embodiments, the computing devicecan be communicably coupled to the binocular viewing assembly 3700 toallow for images to be electronically communicated to at least one ofthe displays viewable through the oculars 3707. In certain embodiments,the computing device can be positioned so that a camera images thedevice and those video images are presented on one of the plurality ofdisplays viewed through the oculars 3707.

In certain embodiments, a surgical visualization system can comprise thebinocular viewing assembly 3700 in addition to one or more displaysconfigured to be viewed by an assistant or other personnel that are notlooking through the oculars 3707. In some embodiments, one or more ofthese displays can be mounted near the surgical site 3730. For example,one or more displays may be mounted to the support structure 3702 or thehousing 3705. In some implementations, the one or more displays can beflat panel displays or images projected from one or more projectorspossibly projecting images on a wall or screen. In some embodiments, atleast one of the one or more displays provides images of the surgicalsite acquired with the stereo microscope camera, endoscope, a retractorcamera, a proximal camera, or any combination of these. In certainembodiments, at least one of the one or more displays is configured toprovide a similar or equivalent view of what is seen through the oculars3707. For example, the display can be configured to present the samevideo images provided on one of the plurality of displays in thebinocular viewing assembly 3700 and may provide the same field of view.As another example, where multiple displays are viewed through theoculars 3707, one display or a combination of displays outside of thebinocular viewing assembly 3700 can be configured to provide the samevideo images viewed on the multiple displays viewed through the oculars3707.

In some embodiments, the additional displays can comprise 1, 2, or 3flat panel displays. The additional display(s) can be arranged at anglesrelative to the oculars 3707, wherein the angle is about ±90 degrees orabout 180 degrees. In some embodiments, the additional display(s) can beconfigured to provide stereo video images. This can allow an assistantor other personnel to view the same or similar stereo video image thatcan be seen through the oculars 3707. In some embodiments, theadditional displays include one or more cameras or other similar devicesconfigured to track the gaze or eyes of a user. This can be used toenhance, tailor, or optimize the view of an assistant or other surgeonby shifting a portion of a display relative to another display or byshifting a film layer(s) above one or more of the displays to achieve adesirable or suitable convergence. This can provide a suitable ordesirable 3D-effect to be rendered.

In some embodiments, the additional displays (e.g., displays mounted onthe housing 3705) have back- or edge-lit LED illumination sources thathave an output of about 1 W, 2 W, 5 W, 6 W, or greater than 6 W of powerto drive the displays in a well-lit surgical environment.

In some embodiments, the additional displays have optics (e.g., lensesand mirrors) to produce a relatively large eye box for each eye. Thiscan be done to allow the assistant or other user some degree of headmotion while still seeing stereo video images. This can also be done toallow the stereo view to be seen at a relatively large distance from thedisplay. For example, stereo video images may be seen at a distance ofabout 5 cm from the display, about 10 cm from the display, about 30 cmfrom the display, or greater than about 30 cm from the display, or anyrange between any of these values. This arrangement facilitates theprimary surgeon positioning the binocular assembly 3700 for theirconvenience and the assistant can still see stereo images from adifferent position (e.g., the other side of the patient and/or table).

In some embodiments, the binocular viewing assembly 3700 includes afiber optic light source or other type of light source. The fiber opticlight source can be configured to provide illumination to one or moreparts of the object being imaged.

In some embodiments, the binocular viewing assembly 3700 includes avirtual user interface system comprising a user interface projector anda user interface camera. The user interface projector can be configuredto project images onto the surgical site. The images can include userinterface elements such as, for example and without limitation, virtualbuttons, icons, thumbnails, arrows, text, or the like. The userinterface camera can be configured to acquire images of the surgicalsite and motions may be made by the user (e.g., the surgeon's hand ortool), wherein the user interface camera acquires images of theprojected images. An image processing system can be operably coupled tothe user interface camera and/or the user interface projector. The imageprocessing system can be configured to determine whether actions takenby a user (e.g., the surgeon or assistant) corresponds to virtualmanipulation of at least one element of the virtual user interface. Insome embodiments, the stereo microscope camera can be used as the userinterface camera.

FIG. 41 illustrates an example binocular viewing assembly 4300 thatincludes surgeon oculars 4307 and assistant oculars 4309. The surgeonoculars 4307 can be configured to provide a view of one or more surgeondisplays 4308 positioned within a housing 4305 of the binocular viewingassembly 4300. The view of the one or more surgeon displays 4307 can beprovided by optical components 4303. Similarly, the assistant oculars4309 can be configured to provide a view of one or more assistantdisplays (not shown) positioned within the housing 4305 of the binocularviewing assembly 4300. The assistant oculars 4309 can be rotatedrelative to the housing 4305 independently of the surgeon oculars 4307.

Example Binocular Viewing Assembly with Displays Above Oculars

FIGS. 38-39 illustrate an example binocular viewing display 3800 havingoculars 3807 beneath corresponding displays 3808. The binocular viewingassembly 3800 includes housing 3805 configured to house displays 3808along with optical components 3803 configured to direct images from thedisplays 3808 to the oculars 3809 for viewing by a surgeon or otheruser.

In this manner, the binocular viewing display 3800 can be configured ina manner similar to a periscope. This allows for the optical path fromthe oculars 3807 to the displays 3808 to be of a suitable length withoutmoving the oculars further from the surgical site. For example, thehousing 3805 can include a contoured portion 3806 configured to allow asurgical microscope camera to be positioned near the housing 3805. Thus,the surgical microscope camera can be positioned near the housing toprovide a surgical microscope view of the surgical site. Advantageously,the surgeon can be positioned near the surgical site while viewing thesurgical microscope view of the surgical site through the oculars 3807.By positioning the displays 3808 above the oculars, the optics,electronics, housing 3805, and the like can be positioned so as to allowthe surgeon to be positioned and to manipulate tools in an ergonomic wayduring surgery, reducing fatigue and discomfort.

The contour 3806 can be configured to allow movement of the surgicalmicroscope camera independent of movement of the binocular viewingassembly 3800. This independent movement can allow the surgeon toposition the binocular viewing assembly 3800 in a way that iscomfortable and ergonomically advantageous while allowing forindependent adjustment of the stereo microscope camera withoutdisruption of the ergonomic positioning of the binocular viewingassembly 3800. For example, this can allow the surgical microscopecamera to be placed in a way that allows the surgeon to position theirbody close to the surgical site. This can improve or enhance theergonomics associated with performing surgery with a surgicalvisualization system.

The binocular viewing assembly 3800 thus configured can provide a numberof advantages. For example, the binocular viewing assembly 3800 canprovide situational awareness the surgeon being able to attend to bothviewing the surgical site through a microscope or viewing system andbeing cognizant of the whole patient, the activities of other alliedhealth personnel, and other medical equipment in the room. Therefore,the binocular viewing assembly 3800 and the other various devicesdescribed herein provide both the ergonomic benefit of decoupled,displayed surgical images from the acquiring cameras and the ability toreadily see much of the patient and operating room.

FIG. 39 illustrates the binocular viewing assembly 3800 with the housing3805 cut away to show the optical components 3803 providing a view ofmultiple displays 3808 for each of a left eye and right eye view throughthe oculars 3807. For example, in the field of view of a single eye ofthe oculars 3807, the electronic displays 3808 can be arranged so thatthe user sees both displays 3808.

Example Contoured Binocular Viewing Assembly

With reference to FIGS. 37A-39, example binocular viewing assemblies3700, 3800 are illustrated that include a contoured housing 3705, 3805.This contoured housing can be configured to allow for independentmovement and adjustments of a binocular viewing assembly and a surgicalmicroscope assembly. With specific reference to FIGS. 37A and 37B, thebinocular viewing assembly 3700 includes housing 3705 with a contouredportion 3706 configured to allow movement of the surgical microscopeassembly 3710. This advantageously decouples movement of the surgicalmicroscope camera assembly 3710 and the viewing assembly 3700.Accordingly, a user (e.g., a surgeon) can position the binocular viewingassembly 3700 in an ergonomically satisfactory way. The user can alsoposition the surgical microscope viewing assembly 3710 to provide thedesired or targeted view of the surgical site 3730. Adjustment of theposition and/or orientation of either assembly can be done without orwith reduced effect on the position and/or orientation of the otherassembly. Thus, adjusting the surgical microscope assembly 3710, forexample, does not require the user to move the binocular viewingassembly 3700.

The housing 3705 can include the contoured portion 3706 that can beconfigured as an indentation on a bottom and distal portion of thehousing 3705. The indentation is configured to allow free movement ofthe surgical microscope camera relative to the viewing assembly. Thiscan allow the user to position the surgical microscope camera assembly3710 to acquire desirable or targeted images of the surgical site 3730.

In some embodiments, the binocular viewing assembly 3700 includeshandles 3709 to facilitate movement of the housing 3705. The housing3705 can be coupled to an arm 3703 a of the support 3702, the armconfigured to allow a user to position and/or orient the housing 3705within a range of positions and orientations. The surgical microscopecamera assembly 3710 that provides surgical microscope views can becoupled to a different arm 3703 b to allow independent movement of thesurgical microscope camera assembly 3710 relative to the housing 3705.The contoured portion 3706 allows the surgical microscope cameraassembly 3710 to be positioned relatively closely to the housing 3705.This can advantageously allow the surgical microscope camera assembly3710 to be positioned above the surgical site 3730 close to the oculars.For example, where the surgical site is directly below the housing 3705,the surgical microscope camera assembly 3710 can be configured to havean optical axis that is nearly parallel to gravity for certainsurgeries.

Advantageously, this can allow the surgeon and/or assistant to positiontheir body close to the surgical site to improve comfort and theergonomics of performing surgery. In some embodiments, the housing 3705is thinner at the bottom than at the top. The contoured portion 3706 canbe formed at the bottom of the housing 3705 with electronic displayspositioned above the contoured portion 3706 within the housing 3705. Insome embodiments, the binocular viewing assembly 3700 includes thehousing 3705 and a plurality of oculars, the plurality of ocularsconfigured to provide views of at least one display in the housing 3705.The plurality of oculars include a left ocular and a right ocularsimilarly disposed left and right with respect to a surgical site so asto coincide with a corresponding left eye camera and right eye camera.The at least one display can be configured to receive output videoimages based on stereo video images produced by the left and right eyecameras. The binocular viewing assembly 3700 can be attached to aviewing arm configured to position the binocular viewing assembly 3700.The left-eye camera and the right-eye camera have optical inputs forreceiving light from the surgical site and the distance from the leftand right oculars to the optical input of the left-eye camera and theoptical input of the right-eye camera is not larger than the size of thehousing 3705 of the binocular viewing assembly 3700. Advantageously,this allows a surgeon to ergonomically perform surgery at a surgicalsite beneath the auxiliary optical assembly (comprising the cameras)while viewing through the plurality of oculars the output video imagesof the surgical site displayed on the at least one display. In certainimplementations, the left-eye camera and the right-eye camera are notcoupled to a direct view surgical microscope. In some embodiments, thebinocular viewing assembly 3700 and associated auxiliary opticalassembly are configured so that a surgeon can ergonomically performsurgery at a surgical site that is lateral to the auxiliary opticalassembly while viewing through the plurality of oculars output videoimages of the surgical site displayed on the at least one display. Incertain implementations, the auxiliary optical assembly is smaller thanthe binocular viewing assembly 3700.

Example Surgical Visualization System with Cameras Rotating aboutCentral Aperture

FIG. 40 illustrates an example surgical visualization system 4000 thatincludes a primary surgeon camera and an assistant camera 4010, each ofthe primary surgeon camera and the assistant camera positioned toacquire images of a surgical site from outside the surgical site. Theassistant camera 4010 can be mounted on a rotating ring 4015 or othersimilar structure. The rotating ring 4015 can be configured to rotateabout an objective lens 4005 or pairs of converging optical trains, theobjective lens 4005 or pairs of converging optical trains being part ofan optical system of the primary surgeon camera. For example, theobjective lens 4005 can be used to form images of the surgical site ontothe image sensor of the primary surgeon camera.

In some embodiments, the objective lens 4005 or pairs of convergingoptical trains is rotatable and is coupled to the ring 4015. In someembodiments, the objective lens 4005 pairs of converging optical trainsis configured to be able to remain stationary while the rotating ring4015 rotates. The assistant camera 4010 can be configured to acquireimages of the surgical site as the optical axes of the assistant camera4010 and the primary surgeon camera can be configured to besubstantially parallel and/or can be configured to be directed to thesame or similar region of the surgical site.

The primary surgeon camera can be configured to have a fixed centralarea with one or more sensors and zoom functionality. The objective lens4005 or pairs of converging optical trains can be positioned in themiddle of the rotatable ring 4015 such that the assistant camera 4010 onthe rotatable ring 4015 rotates around the objective lens 4005. The ring4015 can be manually manipulated and/or powered by an electronic motor.In some embodiments, the assistant camera can pivot in addition to beingrotated by the ring 4015. For example, a pivoting system 4020 can beincluded to allow additional degrees of freedom in the movement of theassistant camera 4010 relative to the objective lens 4005 or pairs ofconverging optical trains. In some embodiments, the surgicalvisualization system includes software to reduce the likelihood thatcables will become entangled upon rotation of the ring 4015.

Example Optical Systems for Binocular Viewing Assemblies

FIGS. 26-29 illustrate example optical systems for use with thebinocular viewing assemblies described herein. The optical systems caninclude a binocular optical system and a display optical system, whereinthe display optical system is configured to generate a real image of thedisplay at the field stop between the display optical system and thebinocular optical system. The binocular optical system is configured togenerate a real image of the field stop at a retina of a user. Theoptical system can be configured to have a magnification proportional tothe ratio of the focal length of the binocular optical system to thedisplay optical system. The magnification of the binocular opticalsystem can be proportional to a ratio of the display diagonal to thefield stop diagonal.

In some embodiments, the optical systems can include a display with anoptical system configured so that an exit pupil of the optical system iswithin an eye of a user viewing the display through the oculars (e.g.,where the oculars form at least part of the optical system). This isreferred to as a “near eye display” or an “immersive display” in FIGS.26-29. A difference between the near eye display and the immersivedisplay is a size of the display. The display in the immersive displaycan be larger than the display in the near eye display.

In some embodiments, the optical systems can include a finite conjugatesection comprising the optical components from the display to the fieldstop. In some embodiments, the optical systems can include an infiniteconjugate section comprising the optical components from the display toa collimated section, and a binocular infinite conjugate sectioncomprising the optical components from the collimated section to beforethe eye of the user. The exit pupil of the display optics can be withinthe collimated section and the entrance pupil of the binocular opticscan be within the collimated section. The apparent field of view can bethe angular extent of the image of the field stop within the eye of theuser looking at the display through the oculars (e.g., wherein theoculars are part of the binocular optical system).

Optical Systems for Surgical Microscope Cameras

In some embodiments, optical components directing images from thesurgical site to the surgical microscope stereo cameras can include afield stop positioned after zoom optics of the optical train. This canbe different from an optical system where the field stop is positionedwithin the zoom optics. Advantageously, the field stop after the zoomoptics positions the field stop nearer the image sensor. As the beam(e.g., ray bundle of light through the optical system) size expands andcontracts, the extent of the beam (or beam cross-section) gets largerand smaller. Placing the field stop after the zoom optics allows theoptical system to be compact, making lenses and image sensors smaller(e.g., the lenses can have a diameter of about 0.25 inches and the imagesensor can be about 0.5 inches across). In addition, this allows foroptical components to be inserted to split the beam into differentspectral components, such as visible, near IR, red, green, blue, UV,yellow, etc. This also allows for filter blocks to be easily added tothe optical train. This can result in a relatively small auxiliaryoptical assembly, for example. This can advantageously allow a surgeonto position the auxiliary optical assembly comprising the opticalsystems described here closer to a viewing assembly (e.g., one or moreof the viewing assemblies described herein). With the auxiliary opticalassembly positioned closer to the viewing assembly, this can allow thesurgeon to position the viewing assembly, the auxiliary opticalassembly, and the surgeon's body closer to the surgical site.Advantageously, this allows the surgeon to perform the surgery moreergonomically and comfortably.

Autoclavable Microscope Head

The surgical visualization systems disclosed herein, such as the systemsthat include a binocular viewing assembly, can include a microscope headmade of materials so that it can be sterilized using an autoclave. Thematerials used in the microscope head can be configured to withstandheats and pressures present in an autoclave. The microscope head can beconfigured to maintain its optical and mechanical characteristics afterthe autoclaving process. For example, the optical systems of themicroscope head can be configured to maintain alignment (e.g., anoptical axis of the optical system does not significantly deflect orchange after undergoing the autoclaving process). Similarly, the opticalsystems of the microscope head can be configured to maintain configuredimaging characteristics (e.g., a focal location of the optical systemdoes not significantly change after undergoing the autoclaving process).The mechanical properties of the microscope head can be configured tonot significantly deteriorate after undergoing the autoclaving process.For example, the structural integrity (e.g., brittleness, deformability,elasticity, rigidity of the materials, etc.) can remain within asuitable tolerance after undergoing the autoclaving process. In someembodiments, the microscope head is resistant to significantdeterioration after at least 100 times through an autoclaving process,after at least 200 times through an autoclaving process, or after atleast 300 times through an autoclaving process. The microscope can bemade of materials or combinations of materials to provide the targetedor desired characteristics. For example, the materials can includepolymers such as, for example and without limitation, polypropylene,polymethylpentene, PTFE resin, polycarbonate, polymethyl methacrylate,and the like. Other materials can include, for example and withoutlimitation, metals, plastics, rubber, and other such materials orcombinations of materials. In some embodiments, the microscope head issealable, to make it autoclavable.

Field of View of Primary and Assistant Displays

As described elsewhere herein, the various surgical visualizationsystems that include a binocular viewing assembly can include one ormore additional displays. The additional displays can be configured tobe viewed through an assistant ocular system or the additional displayscan be positioned on a housing of the binocular viewing assembly, on asupport structure near the binocular viewing assembly, on a wall, orprojected onto a wall or screen. In certain implementations, the fieldof view of video images presented on the displays viewed by the surgeonthrough the binocular viewing assembly is the same as the field of viewof the video images presented on one or more of the additional displays.For example, one or more of the additional displays can present anidentical view as that being provided by the displays viewed by thesurgeon. This can include stereo images and/or monocular images. In someembodiments, where the binocular viewing assembly provides a view ofmore than one display, individual displays of the additional displayscan be configured to present video images having the same field of viewas the displays in the binocular viewing assembly.

Stereo and Spectral Imaging

As described elsewhere herein, the optical systems of the cameras (e.g.the surgical microscope camera) can include a common objective and oneor more image sensors. In certain implementations, a single objectivecan be used for left and right imaging channels, wherein additionaloptics focus light from the single objective onto left and right imagesensors. In some implementations, the optical systems can furtherinclude optical components that split optical paths of light based atleast in part on the spectral composition of the light. For example,optical systems can be configured to allow visible light to pass throughan optical component while this same optical component redirects theoptical axis for light caused by fluorescence (e.g., near infraredlight). This can be used to provide an image acquisition system withfour image sensors, with two image sensors for the left channel and twoimage sensors for the right channel, a first image sensor in the leftchannel being configured to receive light within a first spectral bandand a second image sensor in the left channel being configured toreceive light within a second spectral band. The right channel can besimilarly configured.

This can allow stereo images to be produced for both visible and nearinfrared spectral bands, for example. This can also allow for imagesacquired in different spectral bands to be displayed superimposed (e.g.,either superimposed on a display or superimposed through the use ofmultiple displays and optics, as described herein). For example, adisplay left channel can include a first display configured to displayvideo images acquired with the first image sensor in the left imagesensor channel and a second display configured to display video imagesacquired with the second image sensor in the right image sensor channel.A display right channel can be similarly configured.

FIG. 42 illustrates an example optical system 12200 for providing asurgical microscope view of a surgical site. The optical system 12200can include an objective lens, a zoom lens group 12210, an aperture12209, a turning prism, and a video coupler optical system 12220comprising a focus lens group. The focus lens group can focus images ofthe scene onto the image plane where there is an image sensor 12230.This optical system 12200 can be configured, in some embodiments, asillustrated in FIG. 43 to include at least two beam redirectionelements, or beam deflectors, for each of a left optical path and aright optical path. The optical system can redirect the respectiveoptical paths at least twice to provide a compact envelope for theoptical system. This can allow the camera system, e.g., the surgicalmicroscope camera system, to be compact relative to the binocularviewing assembly. Advantageously, a smaller surgical microscope camerasystem can allow for greater freedom of movement and positioning of thecamera. Thus, a surgeon can position the surgical microscope cameraassembly to achieve a desired or suitable view into the surgical sitewith little or no interference with the position of the binocularviewing assembly. Additionally, the compact size surgical microscopeview camera can provide for increased situational awareness and accessto the surgical site for the surgeon. The smaller size reduces thelikelihood that the surgical microscope will block view or access by thesurgeons hands to the surgical site (unlike a large bulky surgicalmicroscope view camera). With the surgical microscope positioned closerto the viewing assembly, this can allow the surgeon to position theviewing assembly, the surgical microscope, and the surgeon's body closerto the surgical site. Advantageously, this allows the surgeon to performthe surgery more ergonomically and comfortably.

The optical system illustrated in FIG. 43 includes left and rightoptical paths. In some embodiments, at least one of the beam deflectorsis a dichroic element configured to provide a first optical path forlight within a first spectral band and a second optical path for lightwithin a second spectral band. For example, the dichroic beam deflectorcan allow visible light to substantially pass through a redirectionelement while this same redirection element folds the optical axis ofinfrared light 90 degrees relative to the optical path for the visiblelight. The respective first and second optical paths generated at thedichroic beam deflector can each be directed onto respective imagesensors. Accordingly, the optical system can be configured to acquirevideo images of a surgical site using left and right optical paths witheach optical path acquiring light in at least two spectral bands. Thiscan advantageously allow the surgeon to view stereoscopic images in twoor more wavelength bands. This may be useful in visualizing fluorescencewithin a surgical site, for example. The surgeon may also have a visiblewavelength reference view akin to the view seen from a direct viewsurgical microscope view of a patient illuminated with white light. Asillustrated, each of the left and right optical paths can utilize acommon objective lens. The optical paths can include at least tworedirection elements to fold the optical axis of the various opticalpaths (e.g., left and right optical paths, first and second opticalpaths branching from respective left or right optical paths, etc.) sothat the optical axis prior to the image sensors is substantiallyparallel to the optical axis after the objective lens. As illustrated,the edge of the objective lens may be truncated to reduce size and formfactor. Accordingly, in various embodiments, the objective lens does nothave a clear aperture that is circular or rotationally symmetric. Ratherthe clear aperture is wider than tall to accommodate both the left andright channels disposed left and right with respect to each other.

The optical paths can include a zoom lens group positioned after theobjective lens. Each of the left and right optical paths can include azoom lens group. On an image side of the zoom lens group, a redirectionelement or beam deflector can be positioned to bend the optical axisabout 90 degrees. On an image side of the beam deflector, the opticalsystem can include an aperture. On an image side of the aperture, theoptical system can include focusing optics. As indicated by thedouble-sided arrow, in some embodiments, the focusing optics can beconfigured to move along an optical axis that is perpendicular to theoptical axis of the zoom lens group to provide focus adjustment andbring into focus an image. On an image side of the focusing optics, theoptical system can include a dichroic beamsplitter to generate first andsecond optical paths. Light from the first optical path can be directedand focused onto a first image sensor while the second optical path canbe redirected by a beam deflector positioned on an image side of thedichroic beamsplitter. Light from the second optical path can bedirected and focused onto a second image sensor. In some embodiments,the optical axis at the second image sensor is parallel to the opticalaxis at the first image sensor. The optical axis at the first and secondoptical sensors can, in some embodiments, further be parallel to theoptical axis between the objective lens and the first beam deflector.

The optical designs provide for a relatively compact surgical microscopecamera assembly while providing a suitable optical path for acquiringvideo images with left and right channels with each channel acquiringimages of at least two spectral bands. In some embodiments, the surgicalmicroscope camera assembly can weigh less than about 1.5 kg. In someembodiments, the image sensors of the surgical microscope cameraassembly each have a diagonal measurement that is less than about aninch, ⅔ inch, ½ inch, ⅓ inch, or ¼ inch or ranges in between any ofthese values. In some embodiments, the surgical microscope cameraassembly can be separated from the binocular viewing assembly by lessthan about 2 feet, less than about 1.5 feet, or less than about 1 foot.In certain embodiments, movement of the surgical microscope cameraassembly can be decoupled from movement of the binocular viewingassembly. For example, in certain implementations, the surgicalmicroscope camera assembly can move independently of the binocularviewing assembly in the z direction (e.g., directly towards and awayfrom a person looking through the oculars of the binocular viewingassembly). In certain implementations, the surgical microscope cameraassembly can move independently of the binocular viewing assembly in thex direction (e.g., left or right with respect to from a person lookingthrough the oculars of the binocular viewing assembly). In certainimplementations, the surgical microscope camera assembly can pitchand/or yaw independently of the binocular viewing assembly. In variousembodiments, roll of the surgical microscope camera and oculars isrestricted to avoid inducing nausea. In certain embodiments, thesurgical microscope, when in use can be separated from the binocularviewing assembly by a space such as a space of at least 6 inches, 1foot, 1.5 feet, 2 feet, 3 feet, 4 feet, 5 feet, etc. or any rangebetween any of these values. Such an open space can provide forincreased situational awareness and increase access of the surgeon tothe surgical site. For example, the surgeon may have more room to reachtoward the surgical site without hitting large bulky equipment.Similarly, reduction of form factor can create a more open environmentwhere the surgeon has more unobstructed views and increase situationalawareness. This configuration provides the surgeon with clearer paths toview, for example, the patient for example when the surgical microscopecamera is situated above the patient and the surgical site is directlyunderneath.

In various embodiments, the surgical microscope camera and/or thebinocular viewing assembly is not attached to the ceiling of theoperating room.

Switchable Views Between Surgical Microscope Camera and Endoscope Camera

In some embodiments, a surgical visualization system can include one ormore connection ports configured to receive video image data from one ormore sources. For example, the connection port can be configured toreceive input from an endoscope. The surgical visualization system canbe configured to switch between providing video images received from asurgical microscope camera and a camera on an endoscope.

A video coupler can be configured to be part of the surgicalvisualization system, wherein the video coupler is configured to converta device meant for viewing with an eye to a device that is configuredfor acquisition with a camera. The video coupler, for example, can becoupled to an endoscope that is configured to provide images suitablefor viewing with an eye. The video coupler, for example, can beconfigured to acquire images suitable for display from the endoscope. Insome embodiments, the endoscope can be a stereo endoscope having anisocenter and intended for use where the horizon is level. The videocoupler can be configured to acquire stereo images from such anendoscope.

Example Visualization System with Multi-View Switching

FIG. 45 illustrates an example visualization system 4700 with twobinocular display units 4705, 4710 configured for multi-view switching.The electronic visualization system 4700 with multi-view switching isshown with 2 binocular display units 4705, 4710 (e.g., BDU), which havestereo or 3D viewing, with 2D monitors 4715, 4720 (in a two-monitorconfiguration) or 2D monitor 4718 (in a single monitor configuration)shown behind each BDU in the line of sight of two surgeons 4725, 4730.The BDUs can include user interface features 4707, 4712 configured toallow one or both of the surgeons 4725, 4730 to control what isdisplayed within the respective BDU 4705, 4710, on the monitors 4715,4718, and/or 4720, or any combination of these. The user interfacefeatures 4707, 4712 can include buttons, switches, foot pedals, touchscreens, or the like. The user interface features 4707, 4712 can beattached to the BDU 4705, 4710 or they can be removable or separate fromthe BDUs. In certain implementations, the user interface features 4707,4712 comprise a single button on a handle or other feature of therespective BDU 4705, 4712.

For certain surgical procedures, two surgeons can work togethersimultaneously. In one portion of the procedure a first surgeon can takethe lead. In a second portion of the procedure, a second surgeon cantake the lead. For example, when two endoscopic surgeons worksequentially, such as an ENT and Neurosurgeon (e.g., in an endoscopicendonasal transsphenoidal surgery) they may use a single endoscope with2 monitors. The ENT can initiate the case and the Neurosurgeon canassist. Then, when they enter the cranial cavity, the Neurosurgeon canlead and the ENT can assist. In such an example, there is a singleimaging modality which they hand off (e.g., an endoscope view). Themonitors 4715, 4720 can be positioned directly in front of each surgeon4725, 4730 for advantageous viewing. The surgeons themselves can bepositioned on either side of the patient for surgical work.

For certain surgical procedures, two surgeons can use an operating roommicroscope. The surgeons may be positioned at 180 degrees apart from oneanother, e.g., two neurosurgeons in a spine case where they are onopposite sides of the table, or two neurosurgeons at 90 degrees in someskull base surgeries. In either scenario, the surgeon orientation isdictated by the nature of using one imaging modality, e.g., themicroscope, and positioning the surgeons for surgical work. Thevisualization system 4700 can improve this situation by providingcomfortable and/or convenient viewing assemblies and monitors to alloweach surgeon to be positioned comfortably and appropriately during asurgical procedure.

The visualization system 4700 can be used to enhance the capabilities ofother visualization systems by allowing multi-view switching betweenrespective surgeons, where multi-view switching includes the ability toswitch between imaging modalities and/or views within a particularimaging modality. For example, to facilitate a simultaneous endoscopic-and microscopic-like approach to surgery through an integratedvisualization system 4700 with multi-view switching, two binoculardisplay units 4705, 4710 can be positioned in front of respectivesurgeons 4725, 4730. The work space orientation of the two surgeons canbe preserved, if desired, and the viewing can be configured to bedirectly in front of each surgeon in a line of sight familiar to them.In certain implementations, other surgeon positions relative to eachother may be used due to the independence provided by the electronicvisualization system 4700 with multi-view selection. In variousimplementations, two or more imaging modalities can be usedsimultaneously, or alternately, with stereo or 3D capabilities by twosurgeons acting independently or assisting one or the other.

The electronic visualization system 4700 with multi-view switchingallows the assistant or co-surgeon to work independently in the sameimaging modality view as the surgeon, or in an alternative imagingmodality view. The co-surgeon or assisting surgeon can select their ownview and monitor the view of the other surgeon in a picture in picture.In some implementations, the co-surgeon or assisting surgeon can switchto a view of another imaging modality, or to another reference imagemodality, such as a 3D volume data set of pre-surgical images. Suchflexibility advantageously circumvents limits of using a single imagingmodality by two surgeons.

To control switching between views, imaging modalities, and the like,any suitable user interface can be implemented. As a particular,non-limiting example, a single button on a handle of the BDU, afoot-switch, or a control panel, connected in parallel, can be used by arespective surgeon or assistant to control the view of the respectiveBDU and/or the view of the other surgeon to cycle through the viewingoptions, e.g., endoscope, exoscope, camera on a tool, electronicsurgical microscope, or reference pre-surgical imaging.

For example, the default mode can be a single button press cycles theimaging modalities for both surgeons in their respective BDUs. If thereis an exoscope and surgical microscope attached to the system, forexample, pressing the selection button cycles through those two choices.If a Dicom image, for example, is added then each use of the buttonbrings up the next of these three options. In certain implementations,the button (or other user interface element) can be configured toindependently control video displayed within the BDUs 4705, 4712 and/ordisplays. For example, each BDU may include a user interface elementthat controls what is seen on the corresponding display in the BDUand/or the display outside the BDU. As another example, each BDU mayinclude a user interface element that controls what is seen on thedisplay in the other BDU and/or the other display outside the BDUassociated with the user interface element. In this way, an assistant orco-surgeon can control what is displayed to each surgeon. In someembodiments, the output video is the same for both surgeons. In someembodiments, the output video is different for the surgeons. In someembodiments, a single user can control the disparate displays/BDUs todisplay the same video or to display different videos on eachdisplay/BDU.

Additionally, by holding the single button on the handle or foot switchdown for a longer duration, such as 1.5 seconds, for example, a singlebutton control can be made to function as an alternate button. Thiswould facilitate another degree of functionality in image choice in theBDU. Users may configured the system 4700 so that the alternate functionis used to control picture in picture. So that with a simple actuationof the button, the view can cycle through the modalities as full viewswith the next view as a picture in picture in both BDUs. Or in anotherconfiguration, the 1.5 second selection can be configured to control theBDU's independently, allowing one surgeon to use a microscope view andthe other surgeon to use an endoscope view (or other modality).

The views can be cycled by depressing a single button by either surgeon.With a ‘timed’ depression of a single button the view choices can becycled by an individual surgeon, independent of the other. With a‘timed’ depression of a single button the view choices can be displayedas picture in picture. This facilitates dependent and independentmulti-view for multiple surgeons in an electronic visualization system.

Display Mounted Image and Camera Controls

FIG. 46 illustrates an example display unit 4600 with camera controls4615 integrated with a handle 4610. The display 4600 unit can displayone or more images using one or more screens to provide a stereo imagewhich is viewable by the user. The images can be generated by a cameramounted to the same stand that holds the display (e.g., a camera thatprovides a surgical microscope view of the surgical site), by a cameramounted near the patient (e.g., proximal cameras), by a camera placed inthe patient (e.g., an endoscope, a camera on a retractor or on asurgical tool), by a graphical display system such as a computer orpicture archiving and communication system (PACS system), or by anotherimage generating source. The display unit 4600 can include a handlemounted 4605 to the display unit 4600 that allows the user to easilyposition the display unit 4600 in an ergonomic and functional position.The handle 4605 includes a button, switch, or lever 4610 that whenengaged releases a brake. This brake holds the display unit still. Thebrake release button, switch, or lever 4610 is positioned on the handle4610 such that the user can release the brake while holding the handlefirmly and using the handle 4605 to manipulate the display 4600, thenthe user can activate the brake by disengaging the button, switch orlever 4610. In some embodiments, the brake on the surgeon display handleis configured to release a brake that allows the display unit to bemoved. In some embodiments, the brake on the assistant display handle isconfigured to release a brake that allows the assistant display to bemoved but leaves the surgeon display locked in place.

Also included on a control panel attached to the display unit or on ahandle is a series of controls 4615 that affect the image seen by theuser. These controls can be buttons, switches, toggles, dials,joysticks, levers, touch pads, or other devices. The controls 4615 canprovide a number of different functions that affect the image. Onecontrol can switch which image or image set is being viewed by the user,switching for example from the stand mounted camera to a camera placedin the patient, to a PACS system to view the radiology data, to acomputer that displays other pertinent data, or to another image source.

Another set of controls can modify the image that is being displayed.The cameras that are connected to the system may have the ability toadjust zoom, focus, iris diameter, color saturation, or other functions.The controls on the display can be configured to communicate with theseremote cameras to adjust some or all of these functions. For example,there could be a toggle or a set of buttons that adjust the zoom of thecamera in or out, and another set that adjusts the focus of the camera.

Another set of controls can modify the position of the camera byactivating a motor to move the camera. For example, a stand mounted ortable mounted camera could have motors that control the position androtation of the camera. The controls could activate the motors, movingthe camera. The user can thereby be looking at or into the display whilemoving the camera so that the camera can be accurately aimed at the areaof interest.

The display unit 4600 can have multiple control panels 4615. These canbe mounted on multiple handles 4605, for example, one on the right sideand one on the left. For example, as illustrated in FIG. 46, mounted onthe left handle are controls which include buttons A through F andjoystick J. In these embodiments, a similar set of buttons, but in themirror image orientation, are included on the right handle. The controlson the different handles can be different, for example, the left handlecan be used to move the camera and the right handle could be used tozoom and focus the camera. Alternatively, some or all of the controls onthe display unit or the different handles can be duplicated such thateither handle can be used to control a function. Further, a duplicateset or subset of controls can be provided in other locations, such asnear an assistant or in a control panel intended for use by the user'sfoot.

The controls can be configured so that one set of controls can performmultiple functions. For example, a multi-position switch can be used todetermine which image input is displayed in the display. This sameswitch can be configured so that a particular control, say a camera zoomcontrol, only activates the feature of the displayed image generator,for example, only the zoom on the currently activated camera changesposition. In this example, the amount of zoom of any other cameraconnected to the system would be unchanged. To change the zoom of theother cameras, they would need to be activated. It would be possible inthis system to also have some or all of the zooms activated at the sametime, but this would probably be undesirable.

Each individual camera may have a number of operations that need to becontrolled. Some of these operations are zoom, focus, iris diameter,light source intensity, light source wavelength, moving in space in thex, y, or z direction, rotating about the x, y or z axis, activating orremoving a filter, taking a still picture, and other standard cameraoperations. Similar types of operations may be needed to manipulategraphical images from other sources, such as zooming into a radiograph,rotating a reconstructed CT scan model, or paging through the patient'srecords. Further operations may be needed for other image sources sothat the user can navigate the graphical user interface of the imagesource. For the most complicated devices, there may be the desirecontrol a large number of different operations. One method to accomplishthis would be to have a matching number of controls. Some embodimentsare configured to have a method to change what the controller activateddepending upon need. For example, if the camera mounted to the stand wascapable of being moved by a motor forward and back as well as to theleft and to the right, this could be controlled easily by afour-way-joystick. If the same camera could also be panned to the leftand right and tilted up and down, a second four-way-joystick could beused to control that. Another embodiment would be to have a switch thatchanged which circuits were affected by the first four-way-joystick sothat it could control movement in one instance and could controlrotation in a second instance. This same switch could toggle the circuitto another location so that the same four-way-joystick could beconfigured to drive the zoom and focus motors of the camera, or the irisdiameter and light source intensity, or other functions that would bedesired.

FIG. 47 illustrates an example embodiment of an electro-mechanicalcircuit diagram that has multiple switches which control one function.Normally Open switches S3, S4, and S5 are connected in parallel so thatshould at least one of them be closed, Normally Open relay RY2 closes,completing the circuit for Motor M1. If any of the Normally Openswitches S6, S7, or S8 are closed, the circuit is completed in reversepolarity for Motor M1, running it in reverse. The normally closedswitches S1 and S2 open the circuit when the motor has moved its targetto one or the other limit. A potentiometer, R1, is included to adjustthe speed at which the motor runs. Switches S3 through S8 can be locatedin various locations. Each switch S3, S4, and S5 can be located in adifferent location, and another switch from S6, S7, and S8 can belocated in the similar location. For example, S3 can be located next toS6. Switch S4 could one pole of a multi-pole joystick, while S7 is theopposite pole. Switch S5 could be one side of a foot-operated toggle,while switch S8 is the other side of that toggle.

To change which circuit is affected by the switch sets, a double polemulti-position switch can be placed in the circuit between Relay RY2 andSwitches S3, S4, and S5 and between Relay RY1 and Switches S6, S7, andS8. This could be a rotary switch, a linear switch, or a single polerelay activated switch. Multiple multi-position switches could be tiedto the same control knob or button so that changing the switch to aparticular position could both set the image displayed to a specificinput source and set the controls to operate features related to thatimage source. For example, the first position on a slider could movemultiple linear multi-pole switches to a first position. One of theswitches, in this first position, would send the signal from a specificcamera to the display, while others of the switches, in this firstposition, would set some or all of the controls to affect functions onthis first camera, such as zoom, focus, position, rotation, etc. Movingthe slider to a second position would change a first switch to aposition that sends a signal from a second camera to the display whileother switches would set some or all of the controls so that theyaffected functions on this second camera. Additional multi-pole switchescould be used in series to switch whether the control affected onefunction or another, for example, a second multi-pole switch couldchange the control switch to affect zoom in one position, x-position ina second position, pan rotation in a third position, and other functionsin other positions. The setting of each multi-pole switch would controlwhich image generating device was active and which function was beingmanipulated.

In some embodiments, a single pole switch can be used to activate arelay that toggles the multi-pole switch to the next position. This samesingle pole switch can be configured to activate a series of relays inparallel so that the multi-pole switches would not need to be physicallyconnected to the same slider. An algorithm that demonstrates how a smallset of buttons could operate a larger number of functions is shown inFIG. 48.

An alternative embodiment would be to use a microcontroller to simulatethe desired circuitry. A program simulating the motor control circuitand the multi-pole relay switch described above can be installed ontothe microcontroller, which can be connected to the multiple displayinputs and the multiple motors or other controlled features. The programcould follow the same algorithm or an algorithm optimized for amicrocontroller. This could have numerous advantages. For example, themicrocontroller could be programmed so that potentiometer switches orHall Effect switches could be used to control both the speed anddirection of a motor in one setting, while being used purely as anon/off switch in another setting. Further, the simulated multi-poleswitch could be programmed so that it is operated by a button and onlytoggles between active sources. In this example, when operating withthree cameras, a PACS system and a computer connected to the display,the button would toggle between five different inputs, while whenoperating with just two cameras connected, the button would only togglebetween two inputs.

In some embodiments, a medical apparatus is provided that includes astereoscopic display, at least one camera remote to the display that isan image source for the display, and a control panel integral with thedisplay that controls functions on the camera. The control panel can bemounted to a handle attached to the display. The handle can include amethod for releasing brakes on the display stand so that the display canbe manipulated to an ergonomic position using the handle. A secondcontrol panel can be mounted to a second handle. The second controlpanel can be configured to duplicate the controls of the first controlpanel. The second control panel can be configured to operate differentfunctions from the first control panel. The control panel can beconfigured to operate functions on the camera that include one or moreof the following: zoom, focus, iris diameter, light source intensity,light source wavelength, moving the camera in space in the x, y, or zdirection, rotating the camera about the x, y or z axis, activating orremoving a filter, taking a still picture, and other standard cameraoperations. A first control panel can operate a first subset of possiblefunctions. A setting can be changed so that the first control panel canoperate a second subset of possible functions wherein some or all of thefunctions are different than the first subset of functions. Thespecified subset of possible functions can vary depending upon the typeof camera that is providing images to the display. At least oneadditional image source can be included that can supply images to thedisplay. The at least one additional image source can be at least one ofan additional camera, a radiology viewing system, and a computerdisplay. Switching from one image source to another can change thefunctions that the control panel operates. Switching from the firstcamera to an additional camera can be configured to change the controlpanel from operating functions on the first camera to operatingfunctions on the additional camera. Switching from a camera to anothertype of image source changes the controls so that they operate as inputdevices which can navigate the graphical user interface of the imagesource. The control panel includes a control mechanism that allows theuser to switch which image source is being displayed. The control panelincludes a control mechanism that allows the user to switch whichfunctions the control panel can operate. An example of functionalityincludes pushing a button to cycle through available views. For example,switching of images can be accomplished by GUI or by handle mountedcontrol. Pressing the button once provides a view of the surgicalmicroscope view, pressing the button again provides a view of theproximal camera, pressing the button again provides a picture in pictureview, pressing the button again switches the larger and smaller videosin the picture in picture, and pressing the button again returns back tothe surgical microscope view.

The control panel can include at least one control mechanism chosen frombuttons, switches, toggles, dials, joysticks, levers, or touch pads. Thecontrol mechanism can be part of an electro-mechanical circuit thatoperates a function on a camera or imaging device. The control mechanismoperates a relay that is part of an electro-mechanical circuit thatoperates the function. A switch can be used to set whichelectro-mechanical circuit the control mechanism is connected to. Theswitch can be a multi-position switch so that the control mechanism canbe switched among a plurality of electro-mechanical circuits. The switchis a single pole switch that operates a multi-position relay so that thecontrol mechanism can be switched between a plurality ofelectro-mechanical circuits. The control mechanism is connected with amicrocontroller that operates functions on one or more cameras orimaging devices. A second control mechanism is connected with amicrocontroller that operates functions on one or more cameras orimaging devices. The microcontroller is programmed to control theswitching of the image source and on which functions the control paneloperates. The stereoscopic display can be a binocular display device.

Video Coupler

Accordingly, the surgical visualization systems described herein can beconfigured to generate or acquire video images from a variety ofsources. In some embodiments, a surgical visualization system caninclude one or more video couplers that are configured to receiveoptical input from a variety of sources. For example, a video couplercan comprise a camera attachment that has a fixed focal length or thatincludes zoom optics, the camera attachment can also be a mono or stereoconfiguration wherein the camera attachment forms an entrance pupil thatis configured to coincide with an exit pupil of an imaging systemintended to be coupled to the visualization system, such as an endoscopeor an exoscope.

An exoscope can be a device similar to an endoscope whose field of viewand distribution of illumination is narrower than an endoscope andattached to a positionable arm attached to a bed or stand. Typically, anexoscope views a surgical site surgery from outside of the surgicalopening. An example of an exoscope is described in U.S. Pat. No.8,702,602 to Berci et al., entitled “Exoscope,” issued Apr. 22, 2014.For example, an exoscope can serve to observe and illuminate an objectfield on a patient from a position set apart from the patient's body.The exoscope can include a lens system configured to observe the objectfield and an illumination configured to illuminate the object field. Theexoscope can be mounted by using a bracket in such a way that, throughthe lens system, an object field can be observed at a distance of a fewcentimeters, such as in the range of about 20 cm, from the distal lightoutlet or image entry end. Exoscopes may for example include devicesthat are observation instruments based closely on successful invasiveendoscope technology but serving for extracorporeal illumination andobservation of an object field.

For many endoscopes and exoscopes, there is no integrated image sensor.However, these devices generally include exit pupils to be viewed by aneye or to couple to a video coupler. Accordingly, the surgicalvisualization systems disclosed herein can include a video couplerhaving an image sensor and imaging optics configured to receive opticalinformation from an external device (e.g., an endoscope or exoscope)that does not include an imaging sensor to generate video images of thefield of view viewed by the external device. These video images acquiredwith the video coupler can then be used in the way other camera systemsare used in the disclosed surgical visualization systems.

For example, a surgical visualization system can be configured to switchbetween different image sources or cameras. Examples of the camerasinclude proximal cameras, surgical microscope cameras, endoscopes(potentially through the use of a video coupler), retractor cameras,surgical tool cameras, exoscopes (potentially through the use of a videocoupler), and the like. The video acquired with the variety of sourcescan be displayed on one or more displays within a display unit, such asa binocular display unit or other display unit described herein.

In some embodiments, the video coupler is configured to optically coupleto the exit pupil of the endoscope or exoscope wherein the opticalinformation from the endoscope or exoscope may be divided using a prismor beamsplitter or mirror and a lens assembly to form a right eye andleft eye path to a stereo sensor(s) within the video coupler. Such anarrangement can be used to produce stereo views within a display unit.

In a stereo embodiment with visible light and near infrared light, e.g.,for fluorescence imaging, each eye path can be configured to contain adichroic beamsplitter directing light to a sensor, or group of sensors,for each respective waveband. For example, a video coupler containing apair of 3-chip visible cameras can be coupled with one or more sensorsfor acquiring near infrared video.

In a stereo embodiment with visible light and near infrared light, e.g.,for fluorescence imaging, each eye path can be configured to contain adichroic beamsplitter directing light to a group of sensors for eachrespective waveband. For example, a video coupler can contain a pair of4-chip prisms with sensors for RGB and NIR for each eye path.

In a stereo embodiment with visible light and near infrared light, e.g.,for fluorescence imaging, a timing and control system configured toaccept/acquire image information at a portion of a sensor for eachrespective waveband can be implemented. For example, a timing andcommunication system can be configured to start and stop the respectivewavebands of the illumination source or sources. For example, a videocoupler can be configured to include a pair of single sensors with RGBWpixels, with one channel for each eye path.

These example embodiments may be implemented by dividing a sensor orgroup of sensors to a right eye and left eye area, respectively.

In some embodiments, the video coupler can include a notch or blockingfilter for fluorescence excitation. This may be advantageous where theexoscope or endoscope does not include such a filter.

User Control Systems & Control of Image Intensity

As discussed above, there have been a variety of imaging systemsdeveloped to enhance the surgeon's view of the surgical site. Theseimaging systems can include optical surgical microscopes or digitalvideo cameras and display systems. The digital video cameras can be usedas endoscopes, which are placed inside the patient and are used inkeyhole type surgery, or can be used as exoscopes, which are placed ator near the surgical wound to focus on the surgical site.

FIG. 49 schematically illustrates an example of such an imaging system5001 in a simplified operating room configuration. The imaging system5001 includes an image acquisition subsystem 5002 having a digitalcamera 5012 and one or more light sources, such as lights 5014, 5015.The digital camera 5012 and lights 5014, 5015 are focused on an area ofinterest (e.g., a surgical site) of the patient on the table 5011. Theuser (e.g., surgeon) can look into the display 5013 to see a magnifiedview of the surgical site. The imaging system 5001 also includes aremote control 5016 which allows the user to adjust the digital camera5012 as needed.

As discussed above, these imaging systems can incorporate optical ordigital filters to enhance the image. One type of filter can enhance theview using false-color imaging where information resulting from themeasurement of non-visible light is used to produce images in thevisible spectrum. For example, near-infrared light can be measured by adigital sensor, a digital transformation can be applied to the measureddata, and light in the visible spectrum corresponding to light in thenear-infrared spectrum can be presented on the display.

In certain embodiments described herein, an integrated visualizationsystem is provided which advantageously permits switching amongdifferent surgical views to facilitate the surgeon's own viewingoptions, to coordinate the work with assistants or other surgeons, orboth. The integrated visualization system may advantageously use lessspace than would separate visualization systems and may advantageouslyincorporate convenient controls for multiple visualization modalities(e.g., when comparing or alternating between microscopes, endoscopes, orother visualization modalities).

In certain embodiments, a visualization system advantageously allows forviewing and switching between multiple visualization modalities usingonly a single stand or cart with the multiple visualization modalitiesattached via arms or other connections.

In certain embodiments, a visualization system advantageously assists arange of users (e.g., surgeons, assistants) and can be tailored for oneor more approaches or surgical sites. The visualization system can beused by a single surgeon or simultaneously by one or more surgeons orco-surgeons working on the same patient (e.g., three- or four-handedsurgery).

In certain embodiments described herein, a visualization system isprovided that utilizes some or all of the features of surgicalvisualization systems, as disclosed elsewhere herein. In certainembodiments described herein, the visualization system canadvantageously combine certain functionalities and advantages of thesurgical visualization systems disclosed above and elsewhere herein(e.g., utilizing a surgical microscope and an exoscope). For example,the visualization system can include a 3D digital camera that can beplaced near the surgical site (e.g., like a standard exoscope) or can beplaced above the site (e.g., like a surgical microscope). In either ofthese modes, the user can look into a display that provides a 3D view ofthe surgical site, with the display placed in an ergonomic position forthe user, unrelated to the position of the digital camera.

Using these visualization systems, multiple cameras can be placed aboutthe surgical site and can be viewable in the same display. These camerascan include one or more of: a digital microscope, an e-scope, a surgicalmicroscope, an exoscope, an endoscope, or other surgical microscopes orcameras that provide a view of a surgical site. To avoid having todisconnect one camera and connect another to the display each time theuser wants to change views, a video switch can be employed. All videosignals can go to the switch, and the desired video signal can be outputto the display.

Some of the cameras contemplated to be used with the visualizationsystem described herein have adjustments like zoom, focus, iris diameterthat can be controlled. Further, some of these cameras are mounted tosystems that allow adjustment of the camera's position, for example,panning the camera left or right or adjusting the up and down tilt ofthe camera. In certain embodiments described herein, the visualizationsystem can be configured such that these adjustments can be doneremotely via a remote control device (e.g., a panel comprising switches)that send electronic signals to motors or other electronic deviceslocated on or in the camera. As the number of cameras and the number ofadjustable options per camera grows, the number of switches of theremote control device could be expected to increase. At some point,however, the increased number of switches would make ease of operationbecome a challenge. When that happens, the user may decide to look atthe remote control device instead of maintaining the user's view of thesurgical site. Certain embodiments described herein advantageouslyprovide a way to simplify the remote control device so that the user canoperate the remote control device through memory without having to lookaway from the surgical site, yet can retain the ability to operate anydesired functions of the cameras.

In some configurations, the different camera settings may utilizedifferent light conditions to facilitate viewing. The different camerasettings can include different zoom levels, where a tighter zoomutilizes a more focused light and a wider zoom utilizes a more diffuselight. The different camera settings can include different lightspectra. For example, instead of broad spectrum visible light (whitelight), it could be desirable to use blue light or non-visible light,such as near infrared. Some existing surgical visualization systems arealready configured such that changing a camera setting may change alight setting (e.g., changing the focus on some optical microscopeschanges the light intensity and spread). Yet, these systems are notconfigured to work with other cameras or multiple camera visualizationsystems. Further, some cameras may utilize more light than others toachieve the desired quality of visualization. Smaller cameras, withtheir smaller diameter objectives and smaller sensor chips, can oftenutilize more illumination of a subject for the same clarity. Whenswitching from a smaller camera (e.g., an exoscope) to a larger camera(e.g., a digital surgical microscope camera), the intense light utilizedfor the exoscope may appear very bright when the surgical microscopecamera image is shown in the display. This effect could wash out thedigital display in the area that is illuminated such that the worksitecannot be adequately visualized, or worse, this effect could cause flashblindness to the user when switched. Additionally, bright light, whethervisible or non-visible light, can cause warming or even burning of thepatient's tissue. Even though flash blindness may not occur with unusedlights being left on, there are advantages to having the unused lightsdimmed or turned off when not being utilized. Certain embodimentsdescribed herein advantageously provide a system which can adjust thesettings on the integrated lights, for example, automatically toprogrammed settings for the camera selected by the user from which theimage is being displayed. In certain embodiments, when the display ischanged to a different camera view from a different camera selected bythe user, the system can advantageously automatically change thesettings on the integrated lights to the programmed settings for thedifferent selected camera.

When switching among different cameras or adjusting the cameras, otherfactors can affect the lighting to be utilized as well. For example,additional issues that can affect the desired lighting conditions to beused can include camera position, distance to the workspace, and type oftissue being imaged. Because the state of the surgical site can bechanged constantly, when switching to a particular camera, the desiredlighting conditions may have changed. For example, the surgeon may usean endoscope to create a channel for further surgical steps, and maythen use a different camera with a wider field of view to reposition theendoscope in the channel. The desired lighting conditions for the newendoscope position may be different and therefore, when the surgeonswitches back to the endoscope, the image being viewed may be too dark,too bright, washed out, or may have other problems related to thelighting conditions. Certain embodiments described herein advantageouslyprovide a system which adjusts the lighting conditions being utilized inresponse to one or more of: changes of camera position, distance to thearea of interest, type of tissue being imaged, or other conditions.

Viewing in surgical sites can utilize supplemental illumination (e.g.,lighting in addition to room or overhead lighting). An endoscope-likevisualization modality can be selected by the user when the area ofinterest is not directly viewable from outside the body (e.g., lookingoff at an angle behind an anatomical structure or within a body cavity).In such a case, the lighting can be generally directed along the view ofthe endoscope-like device. A microscope-like visualization modality canbenefit from supplemental illumination since an image of a small sitecan be greatly magnified, and the desired amount of illumination oftenincreases as an image is magnified. Certain embodiments described hereinadvantageously provide a system which controls the supplementalillumination in response to the selected visualization modality.

Large changes in illumination levels between modalities can be relatedto the field of view of the camera relative to its working distance. Theintensity of illumination is proportional to the inverse square of thedistance from the light source, which is known as the inverse-square law(e.g., expressed for light intensity as intensity=power/area). Theconsequence of this relationship is that illumination varies with achange in position. For example, when the illumination level of thelight source is held at a constant output level and a change of workingdistance is made, the effect on the resulting image can be far morepronounced when the tissue being viewed is close to the light source,e.g., when the field of view and illumination mapping are fromendoscope-like devices (e.g., wide field of view). Conversely, when thetissue being viewed is far from the tip of the device, the effect on theresulting image from small variations of either the illumination levelor the working distance can be dramatically reduced.

If the illumination emanating from the tip of an endoscope-like device,or the illumination utilized with a microscope or its digitalequivalent, is not held at a constant output level and a change ofworking distance is made, the illumination level at the area of interestas seen by an imaging system can be manipulated to appear similar. Sucha change can utilize the intervention of the user. However, certainembodiments described herein advantageously provide a system which canchange the illumination automatically so that the switching of onemodality to another can be seamless or substantially seamless, relativeto light levels viewed on an electronic display.

In certain embodiments described herein, the camera gain levels can bechanged automatically when the light sources are switched in thevisualization system. By changing the camera gain levels instead ofchanging the illumination, certain embodiments described hereinadvantageously provide a system which reduces the potential tooverexpose tissue to heat, whether the illumination is white light forimaging or near-infrared for excitation. In certain embodiments, theimaging field of view and the illumination field of coverage can beadjusted to be approximately equal, even as magnification changes.

However, the cone of imaging optics and illumination between modalitiescan differ. For example, operating room (OR) microscopes can have conesof imaging optics and illumination that vary between negative angles ofa few degrees to a small number of positive degrees. Typically, the areaviewed on the patient can be smaller than the size of the objective lensof the OR microscope-like device at high magnification. At the lowestmagnification, the area viewed can be at most several times larger thanthe objective lens. OR microscopes and digital equivalents can have longworking distances relative to the area observed. For cameras of thesetypes, a change of 10 mm in distance can represent a small percentagechange in the illumination level. Such long working distances can resultin a shallow depth of field, so the user can find focus and can stay ina standoff position. Typically, OR microscopes are relatively stationary(e.g., held by an arm from a stand), and can be positioned in afavorable viewing position and the surgical work can then proceed.

In contrast, endoscopes or wide angle cameras on tools or retractors canexperience significant movement during operation. Endoscopes and camerason tools and retractors can have wide fields of view (e.g., 70 to 110degrees) and are often hand-held, can be smaller, and can be used muchcloser to the area of interest than can an OR microscope-like device. Asa result, the endoscope or cameras on tools and retractors can bepositioned and repositioned as the surgical procedure is performed toview what the surgeon deems relevant. Endoscopes and cameras on toolsand retractors also can have short working distances relative to ORmicroscopes and working distances that approximate the area observed.The illumination power can vary widely (e.g., to keep the illuminationlevel constant as distance to the subject changes). In particular, forimaging modalities that are considered wide angle (e.g., endoscopes andcameras on tools and or retractors), a change of 10 mm in the workingdistance represents a large percentage change in illumination. Forexample, moving closer by 10 mm from a 25 mm nominal working distancecan yield a nearly 3× increase in illumination level, and moving away 10mm from a nominal 25 mm working distance can yield a nearly 2× decreasein illumination level. This effect can cause viewing issues whenswitching the display to the view of a handheld camera such as anendoscope. Certain embodiments described herein advantageously provide asystem that can properly adjust for the variation in effectiveillumination automatically when the user switches imaging modality.

Near infrared imaging utilizing exogenous dyes can utilize one or moreexcitation sources in the infrared, a wavelength domain not seen by thehuman eye. Excitation sources in the 700-800 nm range are invisible, butsafety for the patient, operator and allied health personnel can beconsidered (e.g., by accounting for excitation levels reaching skin orother tissue). The amount of excitation relative to emission level inthe imaging domain is known as Stokes shift and can be large, evenorders of magnitude larger than ambient levels. Therefore, excitationillumination can be blocked (e.g., using an optical filter) fromreturning to any sensor so that only the fluorescence output or emissionof the dye is received by the cameras.

In addition, the infrared source can generate heating of the patient'stissues. Also, fluorescence quenching, due to prolonged exposure of thedye to the emission source, can make imaging more difficult as time goeson. Certain embodiments described herein advantageously provide a systemand method to pulse the emission such that sufficient photon energy issupplied to the fluorescent dye, but a minimum of energy is transferredthrough radiative heating. A duty cycle where the infrared source iscycled on and off can be used to reduce (e.g., minimize) heating whilemaintaining sufficient visible emission from the dye while at a ratesuch that the excitation response is captured in the camera. Certainembodiments described herein advantageously provide a system and methodin which pulsing of the emission source is performed with the pulsestimed so that the video frame is taken by the camera when the excitationresponse occurs.

FIG. 50A schematically illustrates an example visualization systemcontroller 5100 in accordance with certain embodiments described herein.The visualization system controller 5100 comprises a plurality ofcommunication ports 5110 configured to be operatively coupled to aplurality of image acquisition subsystems 5120. The visualization systemcontroller 5100 further comprises at least one image output port 5130configured to be operatively coupled to at least one image displaysubsystem 5140. The visualization system controller 5100 comprises atleast one user input port 5150 configured to be operatively coupled toat least one user input device 5160. The visualization system controller5100 further comprises at least one circuit 5170 operatively coupled tothe plurality of communication ports 5110, the at least one image outputport 5130, and the at least one user input port 5150. The at least onecircuit 5170 is configured to receive data signals 5122 from theplurality of image acquisition subsystems 5120, to transmit controlsignals 5124 to the plurality of image acquisition subsystems 5120, andto transmit output image signals 5142 to the at least one image displaysubsystem 5140. The at least one circuit 5170 is further configured toreceive at least one first user input signal 5162 and a plurality ofsecond user input signals 5164 from the at least one user input device5160. The at least one circuit 5170 is responsive at least in part tothe received at least one first user input signal 5162 by: selecting animage acquisition subsystem 5120 from the plurality of image acquisitionsubsystems 5120, transmitting the output image signals 5142 to the atleast one image display subsystem 5140 in response to the data signals5122 received from the selected image acquisition subsystem 5120. Incertain embodiments, the at least one circuit is responsive tat least inpart to the received at least one first user input signal by generatingthe control signals 5124 and transmitting the control signals 5124 tothe selected image acquisition subsystem 5120. In certain suchembodiments, the at least one circuit 5170 generates the control signals5124 in response to the at least one first user input signal 5162 andthe received plurality of second user input signals 5164.

Cameras

The plurality of image acquisition subsystems 5120 of certainembodiments are configured to be operatively coupled to thevisualization system controller 5100. For example, the image acquisitionsubsystems 5120 can be in wired communication with the visualizationsystem controller 5100 (e.g., via wired communication ports 5110) and/orin wireless communication with the visualization system controller 5100(e.g., via wireless communication ports 5110). In certain embodiments,the plurality of image acquisition subsystems 5120 comprises a pluralityof cameras 5126. FIG. 50B schematically illustrates a partial view of anexample visualization system controller 5100 operatively coupled to afirst image acquisition subsystem 5120 a comprising a first camera 5126a and to a second image acquisition subsystem 5120 b comprising a secondcamera 5126 b in accordance with certain embodiments described herein.The plurality of cameras 5126 can be selected from the cameras describedherein, including but not limited to: endoscopes, exoscopes, camerasproviding surgical microscope views, digital microscopes that are placedabout the surgical site at various positions and angles, and othersurgical cameras.

In certain embodiments, the camera 5126 generates data signals 5122indicative of an image of the surgical site and provides the datasignals 5122 to the visualization system controller 5100, with thevisualization system controller 5100 responding at least in part to thedata signals 5122 to generate and provide the output image signals 5142to the at least one image display subsystem 5140 (e.g., a display 5140).The camera 5126 may capture three-dimensional (3D) video and the display5140 may be capable of displaying video in 3D. The camera 5126 and thedisplay 5140 can also be capable of displaying alternate color images orvideo, for example, false color images based on infrared light imaging.In certain embodiments, the cameras 5126 are configured to show thesurgical field from different angles, different fields of view ormagnification, or different options such as false color imaging.

In certain embodiments, the cameras 5126 can all be the same type ofcamera, while in certain other embodiments, the cameras 5126 are ofdifferent types. A simple camera can be used which has a fixed focallength, zoom, light level to the image sensor, and spectrum of lightrecorded. The camera 5126 can also be mounted in the desired positionand can be manually moved if the position is to be adjusted. Somecameras can have one or only a few possible adjustments. For example, acamera 5126 can have a focus adjustment, since the surgical works spacedistance from the camera may vary as the operation proceeds. A morecomplex camera 5126 can have adjustment of two or more of: focus, zoom,iris opening, color or filter controls, pan, tilt, or other rotationsabout an axis, and other functions.

One or more cameras 5126 can comprise adjustment mechanisms or systemsconfigured to control various features of the camera 5126, including butnot limited to, position (e.g., panning the camera 5126 left or right oradjusting the up and down tilt of the camera 5126), zoom, focus, irisdiameter. In certain embodiments, the visualization system controller5100 provides the ability to control the features of the cameras 5126(e.g., to adjust the zoom and focus) remotely. For example, the cameras5126 can be motorized and configured to receive the control signals 5124from the visualization system controller 5100 which are generated inresponse at least in part to the first user input signal 5162 and theplurality of second user input signals 5164 from the at least one userinput device 5160 (e.g., a remote control device comprising one or moreswitches which can be activated to adjust these functions). By placingthe camera controls in a location remote from the camera 5126, certainembodiments described herein advantageously provide the ability to doone or more of the following: to avoid jostling the camera 5126 whileadjusting a function of the camera 5126, to avoid blocking other camerasor users while adjusting a function of the camera 5126, to provide theability for the user to look at the display while making the adjustment,and to more easily control the sensitivity of the adjustment. Forexample, having remote control over camera positioning can allow theuser to look at the display 5140 while adjusting the aiming of thecamera 5126 so that the center of the workspace can be placed in thecenter of the displayed image. Any or all of these functions can beincorporated into a camera 5126 through the use of motors or otheractuators adjusting the position of parts of the camera (such as lenses)in accordance with certain embodiments described herein.

FIG. 51 schematically illustrates an example camera 5126 comprisingmotors and gears mounted to adjust pan, tilt (e.g., pitch), and focus inaccordance with certain embodiments described herein. While the examplecamera 5126 schematically illustrated by FIG. 51 only has motors thatoperate three functions, in certain other embodiments, the camera 5126can include other adjustable functions, such as zoom and iris diameter.Other mechanisms, systems, arrangements, or configurations for adjustingthe functions of the camera 5126 are contemplated (e.g., not only makingadjustments via operation of motors and gears). For example, somefunctions may be adjusted electronically (e.g., switching the outputfrom the visible light sensor to an infrared sensor).

The motors and gears are schematically illustrated in FIG. 51 as beingexternal to the camera body, yoke, and mount, in accordance with certainembodiments described herein, and to demonstrate how they may operate.In certain other embodiments these motors and gears can be mountedinternally to the camera body. The camera 5126 can comprise a body 201that houses at least one sensor that detects light.

The camera 5126 can further comprise a lens housing 5202 that holds oneor more optical lenses, a motor 5211, a pinion gear 5212, and a gear5213 mechanically coupled to the lens housing 5202. When the housing5202 is turned, cams can move the lenses in a predetermined path toadjust focus. The motor 5211 can be mechanically coupled to the camerabody and can turn the pinion gear 5212 which, in turn, moves the gear5224. In this way, running the motor 5211 can adjust the focus of thecamera 5126.

The camera 5126 can further comprise a yoke 5203 that is mechanicallycoupled to the camera body and to a camera mount (not shown), a secondmotor 5221 mechanically coupled to the yoke 5203, a pinion gear 5222,and a gear 5223 mechanically coupled to the camera body. The yoke 5203holds the camera 5126 in place. When activated, the second motor 5221can turn the pinion gear 5222 which, in turn, moves the gear 5223,causing the camera 5126 to tilt. The camera 5126 can further comprise athird motor 5231 mechanically coupled to the same camera mount as is theyoke 5203, a pinion gear 5232, and a gear 5233 rigidly connected to theyoke 5203. The third motor 5231 can be rigidly connected to the cameramount, while the yoke 5203 is free to rotate about the axis of the yoke5203. When activated, the third motor 5231 can turn the pinion gear 5232which, in turn, moves the gear, 5233, causing the camera 5126 to pan.

Lights

A well-lit operating field is desirable during surgery. Achieving awell-lit operating field can be a challenge when operating deeply in asmall wound. Multiple light sources can be utilized so that differentparts can be easily visualized. In certain embodiments described herein,the plurality of image acquisition subsystems 5120 can comprise aplurality of light sources 5128. The plurality of light sources 5128 ofcertain embodiments are configured to be operatively coupled to thevisualization system controller 5100 (e.g., in wired communication withthe visualization system controller 5100 via wired communication ports5110 and/or in wireless communication with the visualization systemcontroller 100 via wireless communication ports 5110). The light sources5128 can be selected, for example, from the light sources describedherein.

In certain embodiments, each image acquisition subsystem 5120 cancomprise a camera 5126 and a corresponding light source 5128. Forexample, in accordance with certain embodiments described herein, FIG.50C schematically illustrates a partial view of an example visualizationsystem controller 5100 operatively coupled to a first image acquisitionsubsystem 5120 a comprising a first camera 5126 a and a first lightsource 5128 a. The example visualization system controller 5100 is alsooperatively coupled to a second image acquisition subsystem 5120 bcomprising a second camera 5126 b and a second light source 5128 b. Incertain embodiments, the light source 5128 can be integrated with thecorresponding camera 5126, while in certain other embodiments, the lightsource 5128 can be separate from the corresponding camera 5126.

In certain embodiments, the visualization system controller 5100 isresponsive at least in part to the data signals 5122 from the camera5126, the first user input signal 5162, and/or one or more of theplurality of second user input signals 5164 to generate and providecontrol signals 5124 to the light source 5128. When using a camera 5126,it can sometimes be desirable to have a light source 5128 located nearthe camera 5126 and oriented in a similar direction to the camera 5126.A light source 5128 with such an orientation can be least likely to castshadows that cause visualization problems. Therefore, in certainembodiments, one or more cameras 5126 can each have an associated lightsource 5128.

When using one camera 5126, the light emitted from the light source 5128associated with another camera 5126 may cause unwanted shadows. Also,intense light from a light source 5128 can cause heating of thepatient's tissue, so it can be desirable to turn off a light source 5128during times that the light source 5128 is not needed for the selectedcamera 5126. In certain embodiments, one or more light sources 5128 canemit non-visible spectrums of light (e.g., infrared) which can be usedwith a camera 5126 that can provide false-color images of infrared lightviews of the surgical site. Infrared light can cause tissue heating,although an infrared light source generally would not cause shadows whenviewing the surgical site through a separate visual-light-based camera5126. In certain embodiments, the visualization system controller 5100can advantageously turn off the infrared light source when not in use soas to reduce (e.g., minimize) heating of the patient's tissue andpotentially avoid harm resulting from such heating.

Display

The at least one image display subsystem 5140 can include one or moredisplays as described herein. The at least one image display subsystem5140 of certain embodiments is configured to be operatively coupled tothe visualization system controller 5100 (e.g., in wired communicationwith the visualization system controller 5100 via at least one wiredimage output port 5130 and/or in wireless communication with thevisualization system controller 5100 via at least one wireless imageoutput port 5130). In certain embodiments, the at least one imagedisplay subsystem 5140 can be integrated with the visualization systemcontroller 5100.

In certain embodiments, the image display subsystem 5140 (e.g., display5140) can comprise a binocular display device (e.g., a displayconfigured to display 3D images, such as one or more LDC or LED displaysdisposed in a housing which are viewed through a pair of oculars) or adisplay screen configured to be viewed by a user at a distance from thedisplay screen). In various embodiments, such a display is not a directview display, where an optical path is provided from the ocular throughthe housing to the surgical site. The display 5140 can be configured torespond to the output image signals 5142 received from the visualizationsystem controller 5100 to generate and display an image to be viewed bythe user. In certain embodiments configured for use by multiplesurgeons, each surgeon can have a display, and the surgeon can choosewhich camera image would be fed into the display being used by thatsurgeon.

Remote Control Devices

In certain embodiments, the at least one user input device 5160 (e.g.,at least one remote control device) is configured to generate the firstuser input signal 5162 and the plurality of second user input signals5164, and is configured to be operably coupled to the visualizationsystem controller 5100 so as to transmit the first user input signal5162 and the plurality of second user input signals 5164 to thevisualization system controller 5100. The at least one user input device5160 of certain embodiments is configured to be operatively coupled tothe visualization system controller 5100 (e.g., in wired communicationwith the visualization system controller 5100 via at least one wireduser input port 5150 and/or in wireless communication with thevisualization system controller 5100 via at least one wireless userinput port 5150). The at least one user input device 5160 can compriseone or more selector mechanisms, actuation devices, or other inputdevice components including but not limited to: buttons, toggle buttons,switches, toggle switches, rocker switches, triggers, knobs, dials,relays, joysticks, touchpads, and touchscreens. The at least one inputuser device 5160 can include one or more remote control devices asdescribed herein.

In response to the user manipulating one or more of the selectormechanisms, the at least one user input device 160 can generate andtransmit the first user input signal 5162 and the plurality of seconduser input signals to the visualization system controller 5100. The atleast one user input device 5160 can be integrated with the at least oneimage display subsystem 5140, with the visualization system controller5100, or both. Example user input devices 5160 in accordance withcertain embodiments described herein are also referred to herein asremote control devices 5160.

For example, the remote control device 5160 can comprise one button or apair of buttons configured to run a specified motor of a camera 5126 ina forward direction and a backward direction (e.g., the one button or apair of buttons can be dedicated to activating the appropriate motor forincreasing and decreasing the focal distance of the camera 5126). Theremote control device 5160 can be configured such that, in response tothe user pressing a button, the remote control device 5160 is configuredto generate and transmit appropriate user input signals to thevisualization system controller 5100, which is configured to respond tothese user input signals by generating and transmitting appropriatecontrol signals 5124 to the specified motor of the camera 5126, whichthen moves in accordance with the control signals 5124. In certainembodiments, the button of the remote control device 5160 can beconnected to circuitry (e.g., a relay or control board) of the remotecontrol device 5160 that generates and transmits the user input signalsto the visualization system controller 5100, and the specified motor ofthe camera 5126 can comprise circuitry (e.g., a relay or control board)that activates the motor in response to receiving control signals 5124indicative of the button having been actuated.

For another example, the remote control device 5160 can comprise atoggle button, joystick, or variable level rocker switch that operatesvia the visualization system controller 5100 to increase the focaldistance of the specified camera 5126 when operated in one direction andto decrease the focal distance of the specified camera 5126 whenoperated in another direction. In certain embodiments, the togglebutton, joystick, or variable level rocker switch are configured tooperate via the visualization system controller 5100 to change the focusof the camera 5126 at varying speeds depending upon how far or how hardthe toggle button, joystick, or variable level rocker switch is pushedby the user. Depending upon the number of camera functions to becontrolled, the remote control device 5160 can comprise a plurality ofswitches.

For still another example, the remote control device 5160 can comprise aswitch to be actuated (e.g., moved) by the user for designating which ofthe multiple cameras 5126 is to be used as the video source by providingvideo images to be presented on the display for viewing (e.g., whichinput signals 5122 from the multiple cameras 5126 are to be transmittedvia the plurality of communication ports 5110 to the visualizationsystem controller 5100 to be used as the video source). In response tothe user input signal generated and transmitted by the remote controldevice 5160 due to the switch being activated by the user, thevisualization system controller 5100 can respond to the camera inputsignals 5122 from a selected camera 5126 and can generate output imagesignals 5142 based on the camera input signals from the designatedsource and transmit the corresponding output image signals 5142 to theat least one image display subsystem 5140. Certain such embodiments canadvantageously avoid disconnecting and reconnecting display cables whenswitching among camera views to be displayed.

FIG. 52 schematically illustrates an example remote control device 5160comprising a plurality of selector mechanisms 5165 (e.g., switches 5165a and buttons 5165 b) in accordance with certain embodiments describedherein. The example remote control device 5160 of FIG. 52 comprisesswitches 5165 a and buttons 5165 b configured to allow adjustment ofzoom, focus, pan, tilt, other rotations, iris diameter, and false colorimaging. In certain embodiments (e.g., using the example remote controldevice 5160 of FIG. 52), the remote control device 5160 has sufficientswitches 5165 a and buttons 5165 b so that each function on each camerahas a dedicated switch. The example remote control device 5160 alsocomprises one or more selector mechanisms (e.g., switches 5165 a)configured to transmit the first user input signal 5162 to thevisualization system controller 5100, which responds to the first userinput signal 5162 by selecting the data signals 5122 from a selected oneof the cameras 5126 for use in generating the output image signals 5142transmitted to the display 5140. In this way, certain such embodimentsadvantageously allow the user to toggle the viewed video between thedifferent image acquisition subsystems 5120 (e.g., cameras) that areavailable. In certain such embodiments, besides toggling the selectedcamera, the visualization system controller 5100 further responds to thefirst user input signal 5162 by acting upon only the second user inputsignals corresponding to the selected one of the cameras 5126 andgenerating control signals 5124 (e.g., focus, zoom, etc.) transmitted tothe selected one of the cameras 5126. In this way, certain suchembodiments disable the selector mechanisms corresponding tonon-selected cameras 5126 so that the settings of these cameras 126remain unchanged until selected by the user.

In certain embodiments having a large number of image acquisitionsubsystems 5120, the at least one remote control device 5160 may have anexcessive number of selector mechanisms. For example, if there is aremote control device 5160 dedicated to each camera, the user may needto keep track of where each remote control device 5160 is located or mayneed to break eye contact from the display in order to verify whichremote control device 5160 is being operated. One alternative may be tohave one remote control device 5160 with a large number of selectormechanisms (e.g., as schematically illustrated in FIG. 52) so that allthe cameras functions are accessible on the one remote control device5160. However, this configuration may also be problematic because theuser may break eye contact from the display to find the appropriateselector mechanism, as it would be difficult to memorize the location ofeach switch, button, etc. of the remote control device 5160 when thereare a large number of them. In both configurations, the user may notremember which camera is active in the display when wanting to make anadjustment to the camera since it can be difficult to remember whichswitch operates which function.

In certain embodiments, the remote control device 5160 advantageouslyhas a number of remote control selector mechanisms that are configuredto control the same camera functions (e.g., one or more remote controldevices that have a limited number of easy-to-remember selectormechanisms), and configured to vary the function for which the selectormechanisms modify when activated. For example, the user may prefer toperform a primary adjustment with a dominant hand, but while performingthe surgical operation, may want to perform minor adjustments with theother hand to avoid having to interrupt the surgical operation. Foranother example, the user may want to make the adjustments using aremote control device configured to be used with the foot so that bothhands of the user can be used on the surgical procedure withoutinterruption. In such situations, the user may want to maintain eyecontact on the subject matter shown in the display and not break away tolook at the selector mechanisms (e.g., control switches) mounted on theremote control device 160.

In certain embodiments, the remote control device 5160 advantageouslyhas a limited number of selector mechanisms (e.g., switches) mounted onthe remote control device 5160 that perform the known functions, andeach of one or more of the selector mechanisms is configured to performmultiple functions. In certain embodiments described herein, thevisualization system controller 5100 can respond to the plurality ofsecond user input signals 5164 depending upon the information of thefirst user input signal 5162. Certain such embodiments advantageouslyallow the user to actuate a first selector mechanism which selects thedesired camera function (e.g., selects a desired camera, selects adesired function, or both) that is controlled by activating a secondselector mechanism. For example, the visualization system controller5100 can advantageously determine which camera feed is sent to thedisplay and can set the appropriate relays so that the selectormechanisms (e.g., switches) of the remote control device 5160 operatethe desired functions on only that camera. Certain such embodimentsadvantageously allow the remote control device 5160 to have only theselector mechanisms needed to adjust one camera, and the user can adjustthe camera of interest by operating those selector mechanisms withoutchanging the settings for the other non-selected cameras. In this way,when switching back to a first camera after adjusting a second camera,the settings of the first camera are advantageously unchanged from whenit was previously used.

As described above, a camera 5126 can have a large number of functions,at least some of which can be operated infrequently, while others may beused extensively. For example, zoom, focus, pan, and tilt may beadjusted frequently while the surgeon operates on different parts of theanatomy, while iris control, false color imaging, and other functionsmay be adjusted based upon the operating room conditions, camera type,and type of procedure being performed. To simplify the remote controldevice 5160 so that there are a limited number of selector mechanism(e.g., switches), each selector mechanism can be configured to operatemore than one function. A first selector mechanism can be configured tooperate a primary function, such as zoom, and then, when a secondselector mechanism has been activated, the first selector mechanism canbe configured to operate a secondary function different from the primaryfunction. Multiple levels of functions can be incorporated into a remotecontrol device 5160 with a limited number of selector mechanisms.

FIG. 53a schematically illustrates an example remote control device 5401comprising a plurality of selector mechanisms (e.g., a rocker switch5406 and a plurality of buttons 5402, 5403, 5404, 5405) configured to beoperated by hand in accordance with certain embodiments describedherein. The remote control device 5401 of FIG. 53a can advantageouslyallow adjustment of any of the camera functions. One set of buttons5402, 5403 are configured to provide the first user input signal 5162 tothe visualization system controller 5100 so as to select the camera 5126providing the data signals 5122 to be used by the visualization systemcontroller 5100 to generate the output image signals 5142 that aretransmitted to the display 5140. For example, pushing a first button5402 can switch to the next camera 5126 of the set of available cameras5126, and pushing a second button 5403 can switch to the previous camera5126. Pressing the button 5404 can switch the function of the selectedcamera 5126 that is to be adjusted to the next function of apredetermined list of functions (e.g., from zoom to focus, or focus topan), and pressing the button 5405 can switch the function of theselected camera 5126 that is to be adjusted to the previous function ofthe predetermined list. Rocker switch 5406 can be configured such thatwhen the user pushes on the first surface 5407 of the rocker switch5406, the selected function of the selected camera is operated in afirst direction. When the user pushes on the second surface 5408 of therocker switch 5406, the selected function of the selected camera 5126 isoperated in a second direction opposite to the first direction. Therocker switch 5406 of certain embodiments can be a simple single pole,dual throw rocker, providing second user input signals 5164 to thevisualization system controller 5100 such that the control signals 5124transmitted to the selected camera 5126 control the motor to turn at afixed speed in each direction. In certain other embodiments, otherswitches (e.g., hall effect sensors) can be employed to provide seconduser input signals 5164 to the visualization system controller 5100 suchthat the control signals 5124 transmitted to the selected camera 5126control the speed of the motor based on how hard the user pressed on theswitch. FIG. 53b schematically illustrates an example remote controldevice 5411 comprising a plurality of selector mechanisms (e.g., arocker switch 416 having a first surface 5417 and a second surface 5418and a plurality of buttons 5412, 5413, 5414, 5415), configured similarlyto the example remote control device 5401 of FIG. 53a , but with theswitch and button sizes and locations configured to be operated with theuser's foot instead of by hand.

FIGS. 54A and 54B schematically illustrate an example remote controldevice 5160 configured to be operated by hand in accordance with certainembodiments described herein. FIG. 54a is an isometric view and FIG. 54bis a front view. This remote control device 5160 has multiple functionsfor most of the selector mechanisms (e.g., buttons, switches). Forexample, the button 5501 can be used to increase the zoom of a selectedcamera 5126, while the button 5502 can be used to decrease the zoom ofthe selected camera 5126. Similarly, the button 5503 can be used toincrease the focal distance of the selected camera 5126 while the button5504 can be used to decrease the focal distance of the selected camera5126. The switch 5505 can be a four-way joystick that, when pressed tothe left, makes the selected camera 5126 pan left and, when pressed tothe right, makes the selected camera 5126 pan right. When the switch5505 is moved up, the selected camera 5126 can tilt up, and when theswitch 5505 is moved down, the selected camera 5126 can tilt down. Thebutton 5506 can switch the selected camera 5126 that is used to generatethe output image signal 5142 shown in the display 5140 to the nextcamera 5126 in the list of possible cameras 5126.

In certain embodiments, the button 5507 can be a switch that works likea shift key on a typewriter or computer keyboard, whereby pressing thebutton 5507 (or holding the button 5507 down) can change what functionthe visualization system controller 5100 adjusts when the buttons 5501,5502, 5503, 5504 are pressed or when joystick 5505 is moved. When thebutton 5507 is depressed or not depressed, the remote control device5160 transmits a corresponding first user input signal 5162 to thevisualization system controller 5100. The visualization systemcontroller 5100 responds at least in part to the first user input signal5162 by applying a corresponding interpretation to the second user inputsignals 5164 received from the actuation of the other selectionmechanisms of the remote control device 5160. For example, upon pressingthe button 5507, instead of adjusting the zoom of the selected camera5126, the button 5501 can adjust the iris diameter of the selectedcamera 5126 in one direction and the button 5502 can adjust the irisdiameter of the selected camera 5126 in another opposite direction. Asanother example, upon pressing the button 5507, instead of panning theselected camera 5126 left or right, moving the joystick 5505 left orright can cause the selected camera 5126 to tilt, adjusting the view ofthe horizon on the displayed image. As a further example, upon pressingthe button 5507, pressing the button 5506 can switch the selected camera5126 that is used to generate the output image signal 5142 shown in thedisplay 5140 to the previous camera 5126 in the list of possible cameras5126. In certain other embodiments, the functionality of the buttons5503, 5504 and moving the joystick 5505 up and down can also bedependent on whether the button 5507 is depressed or not.

Controller Circuitry

In certain embodiments, the visualization system controller 5100comprises at least one circuit 5170 (e.g., one or more microprocessors).The at least one circuit 5170 can comprise one or more modules and canbe programmed or configured by software code. The at least one circuit5170 can take a wide variety of forms, including processors,microprocessors, specific-purpose computers, network servers,workstations, personal computers, mainframe computers and the like. Theat least one circuit 5170 can be operatively coupled to other hardwareof the visualization system controller 5100, examples of which includebut are not limited to: a computer-readable memory media, such asrandom-access memory (RAM) integrated circuits and a data storage device(e.g., tangible storage, non-transitory storage, flash memory, hard-diskdrive). It will be appreciated that one or more portions, or all of theat least one circuit 5170 and the software code may be remote from theuser and, for example, resident on a network resource, such as a LANserver, Internet server, network storage device, etc. The software codewhich configures the at least one circuit 5170 and other hardware toperform in accordance with certain embodiments described herein can bedownloaded from a network server which is part of a local-area networkor a wide-area network (such as the internet) or can be provided on atangible (e.g., non-transitory) computer-readable medium, such as aCD-ROM or a flash drive. Various computer languages, architectures, andconfigurations can be used to practice the various embodiments describedherein.

In certain embodiments, the at least one circuit 5170 can comprise aplurality of modules (e.g., circuits). For example, the at least onecircuit 5170 can comprise a data signal module (e.g., circuit)configured to receive the data signals from the plurality of imageacquisition subsystems 5120 and to provide (e.g., generate) the outputimage signals 5142 transmitted to the at least one image displaysubsystem 5140. The at least one circuit 5170 can further comprise acontrol signal module (e.g., circuit) configured to generate andtransmit the control signals 5124 to the selected image acquisitionsubsystem 5120 (e.g., the selected camera, the selected light source, orboth) of the plurality of image acquisition subsystems 5120.

The data signal module can be responsive to the first user input signalto determine which data signals 5122 are to be used to provide theoutput image signals 5142 to the at least one image display subsystem5140. The control signal module can be responsive to the first userinput signal to determine which image acquisition subsystem 5120 (e.g.,which camera 5126 and/or which light source 5128) is selected to receivethe control signals 5124 and to determine which functionality of theselected image acquisition subsystem 5120 is to be adjusted. In certainembodiments, the at least one circuit 5170 comprises an integratedcircuit that includes both the data signal module and the control signalmodule, and well as other circuitry used during operation of thevisualization system controller 5100. In certain other embodiments, theat least one circuit 5170 can be distributed among separate circuits.For example, at least a portion of the at least one circuit 5170 can beintegrated in a common housing with the at least one user input device5160, with the at least one image display subsystem 5140, or both. Incertain embodiments, the at least one circuit 5170 is in a differenthousing spatially separate from that of the at least one user inputdevice 5160, the at least one image display subsystem 5140, or both,while being operatively coupled (e.g., via wired communications orwireless communications) to the at least one user input device 5160 andthe at least one image display subsystem 5140. The at least one circuit5170 can be configured to receive signals (e.g., the data signals 5122,the first user input signal 5162, the plurality of second user inputsignals 5164) and configured to transmit signals (e.g., the controlsignals 5124, the output image signals 5142) via one or more wiredcommunication channels (e.g., over wires physically connected to the atleast one circuit 170) and/or one or more wireless communicationchannels (e.g., Bluetooth, WiFi, IR, or others).

In certain embodiments, the at least one circuit 5170 is configured toallow the user to select which video is being viewed by controlling(e.g., switching) which camera 5126 provides the data signals 5122 usedto provide the output image signals 5142 to the display 5140. The atleast one circuit 5170 can also be used to change which functions willbe active when the switches, buttons, etc. of the remote control device5160 are operated. In certain such embodiments, the remote controldevice 5160 comprises sets of multi-pole switches, while in certainother embodiments, greater versatility can be made available byutilizing the at least one circuit 5170 (e.g., comprising amicrocontroller integrated with the remote control device 5160 andprogrammed to generate the appropriate user input signals). For example,the microcontroller of the at least one circuit 5170 can be programmedsuch that actuating a selector mechanism (e.g., a switch closing acircuit) of the remote control device 5160 activates a process thatsends a control signal 5124 to an appropriate camera 5126, light source5128, or camera mount of the selected image acquisition subsystem 5120(e.g., turning on a relay to operate a motor).

In certain embodiments, the at least one circuit 5170 is configured toaccept a series of user input signals to then initiate an activity. Forexample, a button of the remote control device 5160 can be considered a“second function” button. When the button is pushed, the at least onecircuit 5170 receives a first user input signal 5162 in response towhich the at least one circuit 5170 accepts any other second user inputsignal 5164 as a command to initiate a different function than what wasfirst programmed. For example, the at least one circuit 5170 can beprogrammed such that a primary switch of the remote control device 5160initiates a second user input signal 5164 which commands the at leastone circuit 5170 to move the focus motor forward when pressed, but uponpressing the “second function” button of the remote control device 5160,the at least one circuit 5170 can be programmed to instead respond tothe second user input signal 5164 to move the iris diameter motorforward. In certain embodiments, the at least one circuit 5170 can beprogrammed to switch to the “second function” mode for all selectormechanisms (e.g., buttons, switches) until the “second function” buttonis pressed again, while in certain other embodiments, the at least onecircuit 5170 can be programmed to switch back to the primary functionsonce one secondary function has been performed. Certain embodimentsprovide multiple “second function” buttons, while certain otherembodiments provide a “second function” button that can advance the atleast one circuit 5170 to different sets of secondary functions (e.g.,if there are many additional functions relative to the number of buttonsput on the remote control device 160).

In certain embodiments, instead of using a button as a “second function”button as described above, the button can operate like a “shift” buttonto select between first functions and second functions. The at least onecircuit 5170 can be programmed to note the position of the “shift”button. When the button is in a first position (e.g., open), the atleast one circuit 5170 can respond to user input signals initiated bythe other selector mechanisms (e.g., switches) to generate controlsignals 5124 which operate the first functions, and when the button isin a second position (e.g., closed), the at least one circuit 5170 canrespond to user input signals initiated by the other selector mechanisms(e.g., switches) to generate control signals 5124 which operate thesecond functions. Once the “shift” button is released and returns to thefirst position, the at least one circuit 5170 can respond to user inputsignals initiated by the other selector mechanisms to generate controlsignals 5124 which operate the first functions (e.g., therebyre-enabling the selector mechanisms to perform the first functions). Incertain embodiments, the remote control device 5160 can comprisemultiple “shift” buttons or a mix of “shift” buttons and “secondfunction” buttons. In certain such embodiments, the set of “shift”buttons and/or “second function” buttons is advantageously designed toprovide a beneficial or an optimal configuration for the user. However,for the same reason it can be desirable to limit the overall number ofbuttons, it can also be desirable to limit the number of buttons used toexecute a function.

Example System Configuration

FIG. 55 schematically illustrates an example surgical visualizationsystem utilizing a visualization system controller 5100 in accordancewith certain embodiments described herein. The example surgicalvisualization system comprises a plurality of image acquisitionsubsystems 5120 (e.g., cameras 5614, 5615 and light sources 5616, 5617,5618), at least one user input device 5160 (e.g., hand-operated remotecontrol device 5604 and foot-operated remote control device 5605), atleast one circuit 5170 (e.g., controller 5601 and video switch box5607), and at least one image display subsystem 5140 (e.g., display5609). Separate components are shown schematically in FIG. 55, withlines indicating communication channels for signals to be communicatedbetween the different components. As described herein, the at least onecircuit 5170 can be configured to receive and transmit signals via oneor more wired communication channels (e.g., over wires physicallyconnected to the at least one circuit 5170) and/or one or more wirelesscommunication channels (e.g., Bluetooth, WiFi, IR, or others). Incertain embodiments, the individual components of the visualizationsystem controller 5100 can receive power to operate from these samewires, separate power cables, via batteries contained in the units, orother ways to supply power.

The controller 5601 is configured to receive user input signals 5162,5164 (e.g., signals 5602, 5603, denoted by heavy solid lines) from theremote control devices 5604, 5605 and to transmit control signals 5124(e.g., signals 5612, 5613, denoted by heavy dashed lines) to the propercamera components to operate a particular function and to transmitcontrol signals 5124 (denoted by light solid lines) to the proper lightsources 5616, 5617, 5618. The video switch box 5607 is configured toreceive data signals 5122 (e.g., signals 5610, 5611, denoted by lightdashed lines) from the cameras 5614, 5615, respectively, and to transmitoutput image signals 5142 (e.g., signals 5608, denoted by a curved heavysolid line) to the display 5609.

The controller 5601 is shown in FIG. 55 as a separate component, but incertain other configurations, the controller 5601 can be housed in oneof the other components (e.g., the display 5609, the video switch box5607, or one of the remote control devices 5604, 5605). The controller5601 can comprise solid-state components, a programmable circuit boardcontroller, a more complex programmable device such as a personalcomputer, or a combination of devices.

Upon using one of the remote control devices 5604, 5605 to change thecamera 5614, 5615 used to provide video signals 5608 to the display5609, the controller 5601 can send a signal 5606 to the video switch box5607 to change the video signal 5608 sent to the display 5609. The usercan therefore look into the display 5609 and press a button in a knownlocation without having to look at it to switch which camera 5614, 5615is used as the video source that is viewed in the display 5609. Thevideo signals 5610, 5611 from the cameras 5614, 5615 continuously gointo the video switch box 5607. In addition, while the user is lookinginto the display 5609, the camera that is used as the video source beingviewed can have one of its functions adjusted. The user can operate theproper buttons on the remote control devices 5604, 5605 and thecontroller 5601 can send the signal 5612, 5613 to the selected camera5614, 5615 to be adjusted. Besides setting the video signal to be viewedin the display 5609, the controller 5601 can respond to the signal orsignals 5602, 5603 received from the remote control devices 5604, 5605to allow only the selected camera to be adjusted, with no signals beingsent to any other cameras.

Control of Lights

In certain embodiments, the light sources can be controlled through theat least one circuit 5170 so that only the appropriate light sources areturned on at the appropriate intensity for the selected camera that isbeing used as the video source for the displayed image. As the userswitches which cameras are selected, the light sources can be turned up,down, on, or off, to facilitate imaging while avoiding unduly heatingthe patient's tissue. The at least one circuit 5170 of certainembodiments can also pulse any of the light sources for propervisualization and to synchronize with the camera image.

The at least one circuit 5170 of certain embodiments can also assess theimage from each camera for brightness. With that information, the atleast one circuit 5170 can adjust the illumination intensity and thecamera gain to provide the user with a preferred image quality. Forexample, this assessment and adjustment can be performed each time thedisplay is switched from the image from one camera to another, or can beperformed dynamically as a camera is adjusted (e.g., by the remotecontrol device or by physical manipulation). This brightness adjustmentcan be set to periodically adjust the lighting and camera gain so thebrightness among the images remains relatively constant. For example,the periodic adjustments can be performed automatically at a frequencyof 1 Hz (once per second) or more frequently (e.g., between 1 Hz and 100Hz, between 1 Hz and 10 Hz, between 10 Hz and 100 Hz). In certain otherembodiments, the brightness adjustment can be done selectively (e.g.,with just certain cameras, only when particular camera settings arebeing changed by the user, or when switching between cameras to be usedas the video source for the display). In similar manner, the at leastone circuit 5170 of certain other embodiments can assess the images fromeach camera for other parameters which can be adjusted by periodicallyadjusting the light sources and/or the cameras, either automatically orselectively, such that the other assessed parameters of the imagesremain relatively constant.

As described above, in certain embodiments, the at least one circuit5170 is configured to transmit control signals 5124 to the light sources5128 of the surgical visualization system. For example, as schematicallyillustrated by FIG. 55, the same controller 5601 that switches betweenvideo signals 5610, 5611 used to provide the signals 5608 to the display5609 can also switch which light sources 5616, 5617, 5618 in thesurgical visualization system are turned on and at what brightness. Incertain such embodiments, the controller 5601 can store desired lightsettings (e.g., can have previously stored or preprogrammed lightsettings) for each of the light sources that are part of the system. Theuser can send a signal 5602, 5603 to the controller 5601 to switch thevideo signal 5608 being sent to the display 5609. When the signal isreceived to switch video sources, the light sources can be adjusted tothe desired light settings. For example, in certain embodiments, asingle light source is associated with each camera. When the controller5601 receives a user input signal indicative of selecting one of thecameras as the video source to be displayed, the light source associatedwith that selected camera is turned on and the light sources associatedwith other cameras are turned off. In certain embodiments, the lightsource associated with the selected camera is adjusted to maintain aconsistent brightness or other parameter with that of the image from thepreviously-selected camera.

In certain other embodiments, additional light settings may be desired.For example, visualization may be improved by using a light source thatis not associated directly with any camera (e.g., light source 5616 ofFIG. 55). A high level of brightness of this light source may improvethe image of one camera, while a lower level of brightness of this lightsource may improve the image of another different camera. Also, anancillary light source may aid people who are directly viewing thesurgical site, in which case this ancillary light source may be turnedon as long as it does not cause viewing difficulties (e.g., bright spotsor shadows) for the surgeon using the display. In such cases, dimming orturning off these ancillary lights can be beneficial.

In certain embodiments, the visualization system controller 5100advantageously utilizes the different light sources (e.g., an overheadsurgical light source 5616 and other light sources 5617, 5618 eitherassociated with a camera or placed to illuminate the surgical site)which are each controlled by the at least one circuit 5170 (e.g.,controller 5601). The at least one circuit 5170 can comprise differentlight settings for each of the light sources of the system, particularlyfor the light sources corresponding to each camera, and the at least onecircuit 5170 (e.g., controller 5601) can switch the light settings forthe light sources associated with all cameras each time the video sourceis switched. The user can also use a switch on the remote control device5160 to turn the light sources on or off or to change the intensity of alight source as desired.

In certain embodiments, the preprogrammed light settings for thedifferent light sources can be selected to illuminate at differentlevels depending upon which camera signal is being viewed and thesettings for that camera, and/or the preprogrammed light settings can beselected to place the light sources at the desired levels for a knowncamera condition. For example, if one of the cameras is using a largemagnification, a bright focused light source may be used. A secondcamera, however, may be configured to take in a lot of light. If thelight source is left very bright when switching to the second camera,the brightness can be uncomfortable for the user. By presetting thelight source to decrease intensity when switching from one camera toanother, certain embodiments described herein can avoid such discomfort.In certain embodiments in which false color imaging with non-visiblelight (e.g., infrared light) is being used with one of the cameras, whenswitching from that false color image to a visible light image, thenon-visible light source can be turned off. For example, turning off aninfrared light source when not being used to generate an image beingdisplayed can advantageously reduce the heating of the patient's tissuefrom the infrared light source. When switching back to the infraredcamera, the infrared light source can be turned back on automatically bythe at least one circuit 5170 (e.g., controller 5601).

Sometimes a camera being used as the video source is in a differentcondition than is compatible with the preprogrammed light settings. Forexample, this condition can be due to a camera being manually placed ina position not compatible with the preprogrammed light settings or dueto the camera imaging different types of tissue in a manner that is notcompatible with the preprogrammed light settings. Such conditions maylead to an image that is darker or brighter than is desired, causingviewing issues, discomfort for the user, or distraction from thesurgical procedure being performed. In certain embodiments, the user mayintervene in such conditions to change the brightness, while in certainother embodiments, the visualization system controller 5100 mayautomatically change the brightness.

In certain embodiments, the at least one circuit 5170 is configured toreceive the data signals 5122 from the image acquisition subsystems 5120and to generate control signals 5124 configured to improve the resultingoutput image signals 5142 transmitted to the display 5140. For example,the at least one circuit 5170 can assess the brightness of the imageprovided by the selected camera and can adjust the light source settingsand/or the camera settings to bring the brightness of the image from theselected camera into a desired or predetermined range. Alternatively,the at least one circuit 5170 can be configured to receive the datasignals 5122 from the image acquisition subsystems 5120 and to generateoutput image signals 5142, based on the received data signals 5122, thatare configured to improve the image being displayed on the display 5140.For example, the at least one circuit 5170 can assess the brightness ofthe image provided by the selected camera and can generate output imagesignals 5142 which adjust the brightness of the image to be displayedinto a desired or predetermined range. Certain such embodimentsadvantageously enable the user to view a displayed image while avoidingdistracting or uncomfortable levels of brightness.

In certain embodiments, the at least one circuit 5170 compares one ormore attributes (e.g., brightness) of the image provided by a firstcamera and one or more attributes (e.g., brightness) of the imageprovided by a second camera and automatically adjusts the light sourcesettings and/or the camera settings to bring the displayed images fromthe first camera and the second camera closer to having the sameattributes. Alternatively, the at least one circuit 5170 can compare oneor more attributes (e.g., brightness) of the image provided by the firstcamera and one or more attributes (e.g., brightness) of the imageprovided by the second camera and can generate output image signals 5142based on the received data signals 5122, that are configured to bringthe displayed images from the first camera and the second camera closerto having the same attributes. Certain embodiments advantageously enablethe user to switch the displayed image between images from the first andsecond cameras while avoiding distracting or uncomfortable differencesin attributes (e.g., brightness) between the images.

FIG. 56 schematically illustrates an example surgical visualizationsystem utilizing a visualization system controller 5100 comprising animage analyzer 5701 in accordance with certain embodiments describedherein. The example system of FIG. 56 is similar to that of FIG. 55, butincludes the image analyzer 5701. The image analyzer 5701 can comprise aportion of the at least one circuit 5170. While the image analyzer 5701is shown separately from the controller 5601 in FIG. 56 (e.g., the imageanalyzer 5701 having its own control board that is configured totransmit a signal to the controller 5601). In certain other embodiments,the image analyzer 5701 can be incorporated into the controller 5601(e.g., incorporated into the same circuit board). For example, if usinga full computer processor such as a Raspberry Pi, the video signal andthe switching can all be run in the same processor using one programwith multiple subroutines. In certain embodiments, the image analyzer5701 receives the data signals 5702 from the selected camera, analyzes aselected parameter (e.g., the brightness, intensity, optical power) ofthe image, and transmits information 5703 regarding the selectedparameter to the controller 5601. This information 5703 can be anaverage value of the selected parameter, a peak value of the selectedparameter, or a combination of average and peak values, as well as withother values. The controller 601 then adjusts the selected parameter ofthe displayed image based upon this information 5703. This process canbe repeated as many times as is appropriate to get the selectedparameter to match the user's preferences.

Control of Display Image

FIG. 57 is a flowchart of an example process 5800 for switching theimage acquisition subsystem 5120 (e.g., camera 5126 and light source5128) selected to be the video source for the displayed image inaccordance with certain embodiments described herein. The process 5800comprises switching the lighting system to the desired state, switchingthe connections from the remote controls and switching what is shown inthe display. The switching can comprise a transition from a firstmodality to a second modality (e.g., switching from a microscope-likemodality to an endoscope-like modality, or switching from anendoscope-like modality to a microscope-like modality).

In certain embodiments, switching views among different imagingmodalities in an electronic visualization system can be initiated whenthe system is in one modality and the user chooses the next or desiredmodality by contacting a button, foot switch, or other selectormechanism of a remote control device 5160. For example, upon the useractivating a switch to change to a new camera, the at least one circuit5170 can respond by comparing the desired next view to the current viewfor compatibility of the overall illumination intensity. Such acomparison can be performed in a manner unseen by the user (e.g., usingonly a few milliseconds of the desired next view for the comparison). Ifthe illumination level of the second view is greater than that of thefirst view, the light source level of the second view can be lowered. Ifthe illumination level of the second view is lower than that of thefirst view, the light source level of the second view can be increased.Such a comparison and adjustment can be applied when performing completeview switching or picture-in-picture or multiple views per screen. Incertain embodiments in which full screen switching takes place, thelight source associated with the view that is not displayed can then bedisabled, shut off, or set to a lower power setting. In certainembodiments in which two or more scenes are both displayed concurrently,the associated light sources can remain on.

In certain embodiments, the process can proceed as follows:

-   -   1) Initiate switching, for example, by selecting a next or        desired second modality (e.g., second camera) different from the        present first modality (e.g., first camera) by actuating a        button, foot switch or other selector mechanism of the remote        control device 5160.    -   2) Detect and possibly modify a level of the second modality        before displaying an image of the second modality on the        display. For example, the level of the second modality can be        modified by either varying the illumination output level,        aperture setting, or camera gain of the second modality, and        such adjustments may involve several cycles or iterations before        displaying the image of the second modality.    -   3) Display the image of the second modality (e.g., in full,        partial, or picture in picture view).

Various methods can be used in accordance with certain embodimentsdescribed herein to accomplish apparent light level balancing betweenviews besides changing the illumination levels of the light sources. Incertain embodiments, the f-number of the optics of any of the modalitiescan be varied as the working distance is changed and the light sourceintensity held constant. In certain other embodiments, the f-number ofthe optics of any of the modalities can be held constant as the workingdistance is changed and the light source intensity is varied.

In certain embodiments, the light sources can be set at a constantoutput and attenuated (e.g., using a variable filter or screen dependingon illumination requirements). This attenuation can be controlled by adrive circuit and stepper motor with optional feedback loop for control.The advantage of mechanically or optically attenuating the constantoutput in this manner can be speed of change without color shift. Incertain other embodiments, the light source itself can have its powerraised or lowered to the desired output level. In certain embodiments,the light sources associated with each visualization modality can beoperated in one of these two ways, in a combination thereof, or in otherways.

FIG. 58 is a flowchart of an example process 5900 for analyzing theimage to be displayed and adjusting the brightness of the image to bedisplayed in accordance with certain embodiments described herein. Thedashed arrow of FIG. 58 is indicative of a loop in which the lighting isanalyzed and adjusted until a satisfactory value is reached. FIG. 59 isa flowchart of an example process 5910 for analyzing the image to bedisplayed for peak emission response due to a duty cycle on anon-visible light source in accordance with certain embodimentsdescribed herein.

To have the at least one circuit 5170 (e.g., controller 5601)automatically adapt the lighting to account for the brightness of theimage to be displayed, the at least one circuit 5170 can measure thebrightness of the image, for example, by reading the image displayed orto be displayed and determining a parameter (e.g., the luminance,brightness, intensity, optical power, or other metric) measured for eachpixel in the image. If the average parameter value is higher or lowerthan a preprogrammed target range, the image may be brighter or darkerthan desired. In certain embodiments, the at least one circuit 5170 canadjust the brightness of the different light sources in the system whilethe brightness of the image is read. Once the average brightness of theimage meets a preprogrammed target or range, the at least one circuit5170 can stop adjusting the lighting. When switching from one cameraview to another, the at least one circuit 5170 can read the image fromthe new camera, assess the brightness, adjust the light sources via apredetermined algorithm, then switch to the new view. The at least onecircuit 5170 can continue to display the previous view on the displayuntil the switch occurs, can switch to a blank view (no signal), or canshow a preset image on the display while making the change.

In certain embodiments, the at least one circuit 5170 can determine theaverage brightness of the image to determine whether the image hasbright or dark spots. For example, if there is excessive brightness inthe center of the image and a lot of darkness at the edges, the at leastone circuit 5170 can adjust a focus lens in front of the primary lightsource which can change the focus of the light, diffusing it across thefield of view better. This same technique can be employed if there aremultiple light sources and one portion of the view has significantlydifferent luminance relative to other portions.

In certain embodiments, the at least one circuit 5170 can determine theaverage brightness of the image to dynamically adjust the brightnesswhile the image is being viewed to reduce (e.g., minimize) visualizationproblems. Such dynamic adjustment can be helpful for cameras that aremoved manually where the working distance or type of tissue being imagedchanges dramatically. The at least one circuit 5170 can constantlymonitor the brightness and when the brightness changes away from adesired level, the at least one circuit 5170 can change the lightintensity appropriately.

In certain embodiments, the at least one circuit 5170 can aid in the useof non-visible light, such as infrared, in another way. When viewingfluorescence generated by infrared light, leaving the infrared lightsource on continuously can lead to patient heating issues. In certainembodiments, the at least one circuit 5170 generates and transmitsappropriate control signals 5124 to the infrared light source to pulsethe infrared light source at a rate that synchronizes with the camerabeing used to view the fluorescence. In certain such embodiments, the atleast one circuit 5170 can control the pulsing of the infrared lightsource and can capture the peak emission response.

In certain embodiments, the at least one circuit 5170 can control theswitching of the video source and the light sources, and can serve as acentral camera clock, timecode, record trigger, or synchronizingreference between multiple cameras. The at least one circuit 5170 ofcertain embodiments can also provide synchronization between the camerasand light sources.

Digital video technology, whether recording the video or immediatelytransmitting the video, can be done by capturing “still” images at aconstant rate. A video that is captured with a shutter speed of 1/125s(seconds) at a frame rate of 30 fps (frames per second) can have theimage being captured for 8 milliseconds, then the shutter can be closedfor 25⅓ milliseconds, then another image can be captured for 8milliseconds as another cycle begins. This cycle can be termed the “dutycycle” of the camera. The cycle described is one possible cycle, andsome cameras can have their duty cycles adjusted via settings that canbe controlled.

When the fluorescence response is near instantaneous and potentiallyshort lived, the synchronized cameras can be viewing, their sensorsactive, during the excitation and during the emission pulse. If thecameras are not synched to the duty cycle of the excitation lightsource, the result can appear as dropped frames. To avoid such droppedframes, the duty cycle can be set at a rate so high as to not producevideo frames without excitation. For example, if the camera is runningat 30 fps with a shutter speed of 1/125s, an excitation pulse rate of120 pulses per second would ensure that there is an excitation pulseoccurring every time the shutter is open. In this example, there canalso be approximately three pulses occurring when no image is beingcaptured.

In certain embodiments, the at least one circuit 5170 can be configuredto measure the brilliance of the illumination and to adjust the timingof the light sources so as to adjust (e.g., optimize) the lighting dutycycle to correspond with the digital video capture rate. The at leastone circuit 5170 can set the duty cycle of the emission source to matchin rate the capture rate of the camera. For example, if the camera isset to 30 fps, the emission is pulsed 30 times per second. If the camerais set to 60 fps, the pulse rate will be doubled. The at least onecircuit 170 can then start measuring the brilliance of the imagecaptured on the video sensor (e.g., measuring brilliance of just theimage values in the spectrum of the excitation response). The at leastone circuit 5170 can then adjust the start time of the duty cycle of theemission source. When the timing of the duty cycles are matched, themaximum brilliance can be measured. The at least one circuit 5170 canthen adjust the duty cycle of the emission source, changing the lengthof time that the source is emitting versus the length of time that thesource is off. The at least one circuit 5170 can select the shortesttime that the source is on that provides an increased (e.g., maximal)brilliance, thereby reducing (e.g., minimizing) the amount of lightexposed to the patient without decreasing the fluorescence imagingquality.

By having control over the illumination sources in the lighting system,the visualization system controller of certain embodiments describedherein can generate the pulsed illumination for properly visualizing thefluorescence. The at least one circuit 5170 can also turn on, off, ordim the illumination sources in the lighting system to meet theparameters of the various cameras in the system. In certain embodiments,the at least one circuit 5170 can incorporate a feedback loop to assessthe image brightness and color qualities, and can modify the lightsources or adjust the camera settings to provide a preferredvisualization of the workspace.

Headings are used throughout this application as an organizational aidfor the reader. These headings may group together examples of methods,apparatuses, and structures that may generally relate to a particulartopic noted in the headings. It will be appreciated, however, that whilethe various features discussed under the heading may relate to aparticular topic, the headings should not be understood to indicate thatthe features discussed under a given heading are limited inapplicability only to the topic or topics that listed in the heading.For example, a heading may be labeled “Remote Control”. However, thesubject matter included under this heading may equally be applicable tosubject matter contained in any other section, such as content under theheading “Control of lights,” “Example System Configurations,” ControllerCircuitry,” “Pair of Mobile Devices,” and other sections. Alternatively,subject matter from other sections may also be applicable to the “RemoteControl” sections.

Although described above in connection with particular embodiments ofthe present invention, it should be understood the descriptions of theembodiments are illustrative of the invention and are not intended to belimiting. Various modifications and applications may occur to thoseskilled in the art without departing from the true spirit and scope ofthe invention.

Conditional language, such as, among others, “can,” “could,” “might,” or“may,” unless specifically stated otherwise, or otherwise understoodwithin the context as used, is generally intended to convey that certainembodiments include, while other embodiments do not include, certainfeatures, elements and/or steps. Thus, such conditional language is notgenerally intended to imply that features, elements and/or steps are inany way required for one or more embodiments or that one or moreembodiments necessarily include logic for deciding, with or without userinput or prompting, whether these features, elements and/or steps areincluded or are to be performed in any particular embodiment.

It should be emphasized that many variations and modifications may bemade to the above-described embodiments, the elements of which are to beunderstood as being among other acceptable examples. All suchmodifications and variations are intended to be included herein withinthe scope of this disclosure and protected by the following claims.

CONCLUSION

Various modifications to the implementations described in thisdisclosure may be readily apparent to those skilled in the art, and thegeneric principles defined herein may be applied to otherimplementations without departing from the spirit or scope of thisdisclosure. Thus, the claims are not intended to be limited to theimplementations shown herein, but are to be accorded the widest scopeconsistent with this disclosure, the principles and the novel featuresdisclosed herein.

Certain features that are described in this specification in the contextof separate embodiments also can be implemented in combination in asingle embodiment. Conversely, various features that are described inthe context of a single embodiment also can be implemented in multipleembodiments separately or in any suitable sub-combination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asub-combination or variation of a sub-combination.

What is claimed is:
 1. A visualization system comprising: a plurality ofcommunication ports configured to be operatively coupled to a pluralityof cameras and a lighting system comprising one or more light sources;at least one image output port configured to be operatively coupled toat least one image display subsystem; at least one user input portconfigured to be operatively coupled to at least one user input device;and at least one circuit operatively coupled to the plurality ofcommunication ports, the at least one image output port, and the atleast one user input port, the at least one circuit configured toreceive data signals from the plurality of cameras, to transmit controlsignals to the lighting system, and to transmit output image signals tothe at least one image display subsystem, the at least one circuitfurther configured to receive at least one first user input signal fromthe at least one user input device, the at least one circuit responsiveat least in part to the received at least one first user input signalby: selecting a camera from the plurality of cameras, transmitting theoutput image signals to the at least one image display subsystem inresponse to the data signals received from the selected camera, andgenerating the control signals and transmitting the control signals tothe lighting system, the lighting system configured to respond to thecontrol signals by switching light intensity settings of the lightingsystem automatically based on the selected camera to maintain aconsistent brightness of the image to be displayed with that of thedisplayed image from the previously-selected camera.
 2. Thevisualization system of claim 1, wherein the plurality of cameras isconfigured to generate the data signals indicative of an image of asurgical site, the at least one image display subsystem comprises adisplay configured to present an image of the surgical site to a user inresponse to the output image signals, the at least one user input devicecomprises a remote control device configured to generate the first userinput signal.
 3. The visualization system of claim 2, wherein the atleast one circuit is within a housing of the at least one image displaysubsystem or within a housing of the at least one user input device. 4.The visualization system of claim 2, wherein the at least one circuitgenerates the control signals in response to the received at least onefirst user signal.
 5. The visualization system of claim 1, wherein theplurality of cameras comprises at least one endoscope camera and atleast one surgical microscope camera.
 6. The visualization system ofclaim 1, wherein the plurality of cameras is responsive to the controlsignals by varying one or more features of the plurality of cameras, theat least one circuit further configured to respond to the received atleast one first user input signal being in a first state by generatingcontrol signals which vary a first set of features of the plurality ofcameras and to respond to the received at least one first user inputsignal being in a second state by generating control signals which varya second set of features of the plurality of cameras.
 7. Thevisualization system of claim 1, wherein a camera of the plurality ofcameras is associated with a light source, wherein the camera isresponsive to the control signals by varying one or more features of thecamera and the light source is responsive to the control signals byvarying one or more features of the light source.
 8. The visualizationsystem of claim 1, wherein the at least one circuit is furtherconfigured to calculate a difference between an attribute of a firstimage provided by a first camera of the plurality of cameras and to theattribute of a second image provided by a second camera of the pluralityof camera, the at least one circuit further configured to respond to thedifference by generating and transmitting control signals to at leastone of the first camera and the second camera to vary one or morefeatures of the camera to reduce the difference.
 9. The visualizationsystem of claim 1, wherein the light intensity settings include havingat least one of the light sources on and at least one of the lightsources off.
 10. The visualization system of claim 1, wherein thelighting system has programmed light intensity settings that includecycling at least one of the light sources rapidly between off and on togenerate an apparent intensity related to a ratio of time the at leastone light source is on versus time the at least one light source is off.11. The visualization system of claim 1, wherein the light intensity ofat least one light source is variable.
 12. The visualization system ofclaim 1, wherein at least one light source generates non-visible light.13. The visualization system of claim 12, wherein the non-visible lightis in the infrared spectrum.
 14. The visualization system of claim 12,wherein the non-visible light has a wavelength from about 700 nm toabout 1000 nm.
 15. The visualization system of claim 12, wherein thenon-visible light is in the near infrared spectrum.
 16. Thevisualization system of claim 12, wherein the non-visible light has awavelength from about 700 nm to about 800 nm.
 17. The visualizationsystem of claim 12, wherein the non-visible light is in the ultravioletspectrum.
 18. The visualization system of claim 12, wherein thenon-visible light has a wavelength from about 10 nm to about 400 nm. 19.The visualization system of claim 12, wherein the at least one circuitcycles at least one non-visible light source between off and on.
 20. Thevisualization system of claim 19, wherein the at least one circuitcycles at least one non-visible light source between off and on at arate that is synchronized to the image capture rate of the selectedcamera.
 21. The visualization system of claim 19, wherein the at leastone circuit cycles at least one non-visible light source between off andon at a rate such that the fluorescence response is captured by theselected camera.
 22. The visualization system of claim 12, wherein theat least one circuit cycles at least one non-visible light sourcebetween low and high.
 23. The visualization system of claim 1, whereinthe at least one circuit allows for adjustment of the light intensitysettings.